SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 146 Go

enhance: [ExternalTable Part10] enable function output fields on external collections (#49307)

issue: #45881

## Summary

Part 10 of the External Table series. Enables BM25 / MinHash /
TextEmbedding function output fields on external collections, and makes
external-table text_match work with persisted text indexes.

Related: [#45881](https://github.com/milvus-io/milvus/issues/45881)
(External Collection Lakehouse Integration tracking).

### What's in this PR

**Schema & segcore**
- Allow `Function` declarations on external schemas; function-output
  fields skip `external_field_mapping` validation
- `Schema` tracks function-output field IDs; `ChunkedSegmentSealedImpl`
  and `ManifestGroupTranslator` resolve function-output columns by field
  name
- Fast paths for retrieve / search load function-output columns by field
  name; external columns continue using `external_field_mapping`
- `Util.cpp::GetFieldDatasFromManifest` resolves function-output columns
  by field name so indexbuilder reads the correct packed column

**Refresh pipeline (format-agnostic via loon FFI)**
1. `CreateSegmentManifestWithBasePath` creates the input manifest that
   references external original files as CG0
2. `FFIPackedReader(v1)` streams input columns into `InsertData` via the
   shared `ArrowRecordToInsertData` helper
3. `embedding.RunAll` populates function-output fields in-place
4. `FFIPackedWriter` writes output fields as a new column group on top
   of v1
5. `AddStatsToManifest` registers BM25 stats into the final manifest
6. Memory peak stays around one Arrow batch, defaulting to 64 MiB

**RootPath isolation**
- Function-output input manifests, output manifests, BM25 stats, and
text
index artifacts now use the same
`RootPath/insert_log/<collection>/<partition>/<segment>`
  layout as normal external table StorageV3 manifests
- This avoids writing function-output artifacts under bucket-root
  `external/...` paths when multiple clusters share the same bucket

**Text match**
- External text fields with `enable_match` persist text index artifacts
  during refresh and load them through the regular external segment path
- English and Chinese analyzer coverage are both included in the E2E
  test

**Cross-bucket**
- `NewPackedFFIReaderWithManifest` accepts `ExternalReaderContext` and
  injects per-collection `extfs.{collID}.*` aliases when
`external_source` is set. Existing callers pass `{}` and are unaffected

**VirtualPK**
- `VirtualPKChunkedColumn` implements `GetChunk` / `GetAllChunks` so
  proxy requery filter-by-PK works for external collections with virtual
  primary keys

**Shared embedding runner**
- New `internal/util/function/embedding/runner.go` provides canonical
  `RunAll(ctx, schema, data, opts)` shared by import and external-table
  refresh paths
- Supports BM25 output vector types including Float, BFloat16, Float16,
  Binary, and Sparse

### Test Plan

- [x] Go unit tests for `internal/datanode/external` with 99.8% package
  coverage
- [x] Go unit tests for changed import, packed, embedding, and schema
  helpers from the existing branch validation
- [x] C++ rebuild + `milvus_storage` / `milvus_core` lib install from
  the existing branch validation
- [x] E2E function-output tests in
  `tests/go_client/testcases/external_table_function_test.go`
    - `TestExternalTableBM25Function`
    - `TestExternalTableMinHashFunction`
    - `TestExternalTableTextEmbeddingFunction`
- [x] `TestExternalTableTextMatch` with 10 files x 5000 rows = 50000
  rows, English and Chinese text fields, persisted text index object
  checks in MinIO, load readiness check, and query correctness checks

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
W
wei liu committed
7bdd40d6920d1d7a4c789abb98dcbe27d87d5e3f
Parent: 50aca58
Committed by GitHub <noreply@github.com> on 5/28/2026, 2:54:14 AM