SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 148 Go

enhance: Add V3 entry streaming reader (#50073)

issue: #48957

- Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and
`HasEntry` for V3 entries.
- Stream plain and encrypted entries through ordered slices with
incremental CRC32c verification.
- Bound transient stream memory with a process-wide entry stream budget.
- Add dynamically refreshable `common.entryStream.streamBudgetRatio` to
tune the shared entry stream budget for current scalar index loading and
later field data loading.
- Use 16MB stream slices by default, aligned with the encrypted slice
size and existing V3 range size.
- Derive the entry stream budget from CPU core count: `CPU cores *
streamBudgetRatio * 16MB`.
- Harden stream error handling:
  - keep active futures bounded by the thread pool size
  - reject zero or too-small explicit slice sizes
  - use overflow-safe slice count calculation
  - verify CRC for empty entries
  - release budget when callbacks throw or task submission fails
  - wait for active workers before returning on abnormal exits
- validate decrypted slice length and include actual/expected CRC in
stream errors
- Document V3 streaming read behavior and shared budget configuration.

---------

Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
S
sparknack committed
d5917c265e224f4775d122d0aa88556216247622
Parent: 553d402
Committed by GitHub <noreply@github.com> on 5/30/2026, 8:20:13 AM