SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 146 Go

enhance: extend arrow IO thread pool config to DataNode init (#50099)

issue: #50098

## Summary

- Apply `common.arrow.ioThreadPoolCoefficient` /
`ioThreadPoolMaxCapacity` to DataNode at startup, mirroring the
QueryNode wiring from #49208/#49623. Without this, the paramtable keys
exist but have no effect on DataNode and the Arrow IO pool stays at
Arrow's hard-coded default of 8.
- Also call `initcore.InitArrowReaderConfig` so the parquet reader
range-coalescing limits (`hole`/`range` size) apply.
- Register paramtable watchers so capacity changes hot-reload without
restart, matching QueryNode behavior.
- Raise `dataCoord.import.fileNumPerSlot` default from 1 to 4 to reduce
per-import slot demand by 4x and leave more arrow IO pool headroom for
concurrent compactions on the same worker.

## Why

See #50098 for the full reproducer and analysis. Briefly: production
observed a 1.5M-row sort compaction taking >1 hour while the worker's
CPU sat at 12% and network at 14% of baseline — all 5 large
SortCompactions on different pods finished at nearly identical wall
times (variance < 1%), the classic shared-pool saturation signature.
Local bench at 50ms RTT with 100 concurrent readers:

| ARROW_IO_THREADS | conc=100 wall_ms | per-task p50_ms |
|---|---|---|
| 8 (default) | 36,917 | 35,908 |
| 32 | 18,174 | 16,074 |
| 64 | 17,320 | 15,702 |

## Test plan

- [x] `go test -count=1 -run TestComponentParam ./util/paramtable/` (pkg
module)
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
TestRegisterArrowIOThreadPoolWatchers ./internal/datanode/index/`
- [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run
"TestImport" ./internal/datacoord/`
- [ ] CI

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C
cai.zhang committed
4fa49058c1110a0b5e5351ba63320e12e3067698
Parent: e0ae33a
Committed by GitHub <noreply@github.com> on 6/1/2026, 9:08:16 AM