enhance: Add V3 entry streaming reader (#50073)
issue: #48957 - Add `IndexEntryReader::ReadEntryStream`, `GetEntrySize`, and `HasEntry` for V3 entries. - Stream plain and encrypted entries through ordered slices with incremental CRC32c verification. - Bound transient stream memory with a process-wide entry stream budget. - Add dynamically refreshable `common.entryStream.streamBudgetRatio` to tune the shared entry stream budget for current scalar index loading and later field data loading. - Use 16MB stream slices by default, aligned with the encrypted slice size and existing V3 range size. - Derive the entry stream budget from CPU core count: `CPU cores * streamBudgetRatio * 16MB`. - Harden stream error handling: - keep active futures bounded by the thread pool size - reject zero or too-small explicit slice sizes - use overflow-safe slice count calculation - verify CRC for empty entries - release budget when callbacks throw or task submission fails - wait for active workers before returning on abnormal exits - validate decrypted slice length and include actual/expected CRC in stream errors - Document V3 streaming read behavior and shared budget configuration. --------- Signed-off-by: Shawn Wang <shawn.wang@zilliz.com>
S
sparknack committed
d5917c265e224f4775d122d0aa88556216247622
Parent: 553d402
Committed by GitHub <noreply@github.com>
on 5/30/2026, 8:20:13 AM