enhance: [ExternalTable Part10] enable function output fields on external collections (#49307)
issue: #45881
## Summary
Part 10 of the External Table series. Enables BM25 / MinHash /
TextEmbedding function output fields on external collections, and makes
external-table text_match work with persisted text indexes.
Related: [#45881](https://github.com/milvus-io/milvus/issues/45881)
(External Collection Lakehouse Integration tracking).
### What's in this PR
**Schema & segcore**
- Allow `Function` declarations on external schemas; function-output
fields skip `external_field_mapping` validation
- `Schema` tracks function-output field IDs; `ChunkedSegmentSealedImpl`
and `ManifestGroupTranslator` resolve function-output columns by field
name
- Fast paths for retrieve / search load function-output columns by field
name; external columns continue using `external_field_mapping`
- `Util.cpp::GetFieldDatasFromManifest` resolves function-output columns
by field name so indexbuilder reads the correct packed column
**Refresh pipeline (format-agnostic via loon FFI)**
1. `CreateSegmentManifestWithBasePath` creates the input manifest that
references external original files as CG0
2. `FFIPackedReader(v1)` streams input columns into `InsertData` via the
shared `ArrowRecordToInsertData` helper
3. `embedding.RunAll` populates function-output fields in-place
4. `FFIPackedWriter` writes output fields as a new column group on top
of v1
5. `AddStatsToManifest` registers BM25 stats into the final manifest
6. Memory peak stays around one Arrow batch, defaulting to 64 MiB
**RootPath isolation**
- Function-output input manifests, output manifests, BM25 stats, and
text
index artifacts now use the same
`RootPath/insert_log/<collection>/<partition>/<segment>`
layout as normal external table StorageV3 manifests
- This avoids writing function-output artifacts under bucket-root
`external/...` paths when multiple clusters share the same bucket
**Text match**
- External text fields with `enable_match` persist text index artifacts
during refresh and load them through the regular external segment path
- English and Chinese analyzer coverage are both included in the E2E
test
**Cross-bucket**
- `NewPackedFFIReaderWithManifest` accepts `ExternalReaderContext` and
injects per-collection `extfs.{collID}.*` aliases when
`external_source` is set. Existing callers pass `{}` and are unaffected
**VirtualPK**
- `VirtualPKChunkedColumn` implements `GetChunk` / `GetAllChunks` so
proxy requery filter-by-PK works for external collections with virtual
primary keys
**Shared embedding runner**
- New `internal/util/function/embedding/runner.go` provides canonical
`RunAll(ctx, schema, data, opts)` shared by import and external-table
refresh paths
- Supports BM25 output vector types including Float, BFloat16, Float16,
Binary, and Sparse
### Test Plan
- [x] Go unit tests for `internal/datanode/external` with 99.8% package
coverage
- [x] Go unit tests for changed import, packed, embedding, and schema
helpers from the existing branch validation
- [x] C++ rebuild + `milvus_storage` / `milvus_core` lib install from
the existing branch validation
- [x] E2E function-output tests in
`tests/go_client/testcases/external_table_function_test.go`
- `TestExternalTableBM25Function`
- `TestExternalTableMinHashFunction`
- `TestExternalTableTextEmbeddingFunction`
- [x] `TestExternalTableTextMatch` with 10 files x 5000 rows = 50000
rows, English and Chinese text fields, persisted text index object
checks in MinIO, load readiness check, and query correctness checks
Signed-off-by: Wei Liu <wei.liu@zilliz.com> W
wei liu committed
7bdd40d6920d1d7a4c789abb98dcbe27d87d5e3f
Parent: 50aca58
Committed by GitHub <noreply@github.com>
on 5/28/2026, 2:54:14 AM