enhance: extend arrow IO thread pool config to DataNode init (#50099)
issue: #50098 ## Summary - Apply `common.arrow.ioThreadPoolCoefficient` / `ioThreadPoolMaxCapacity` to DataNode at startup, mirroring the QueryNode wiring from #49208/#49623. Without this, the paramtable keys exist but have no effect on DataNode and the Arrow IO pool stays at Arrow's hard-coded default of 8. - Also call `initcore.InitArrowReaderConfig` so the parquet reader range-coalescing limits (`hole`/`range` size) apply. - Register paramtable watchers so capacity changes hot-reload without restart, matching QueryNode behavior. - Raise `dataCoord.import.fileNumPerSlot` default from 1 to 4 to reduce per-import slot demand by 4x and leave more arrow IO pool headroom for concurrent compactions on the same worker. ## Why See #50098 for the full reproducer and analysis. Briefly: production observed a 1.5M-row sort compaction taking >1 hour while the worker's CPU sat at 12% and network at 14% of baseline — all 5 large SortCompactions on different pods finished at nearly identical wall times (variance < 1%), the classic shared-pool saturation signature. Local bench at 50ms RTT with 100 concurrent readers: | ARROW_IO_THREADS | conc=100 wall_ms | per-task p50_ms | |---|---|---| | 8 (default) | 36,917 | 35,908 | | 32 | 18,174 | 16,074 | | 64 | 17,320 | 15,702 | ## Test plan - [x] `go test -count=1 -run TestComponentParam ./util/paramtable/` (pkg module) - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run TestRegisterArrowIOThreadPoolWatchers ./internal/datanode/index/` - [x] `go test -tags dynamic,test -gcflags="all=-N -l" -count=1 -run "TestImport" ./internal/datacoord/` - [ ] CI 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C
cai.zhang committed
4fa49058c1110a0b5e5351ba63320e12e3067698
Parent: e0ae33a
Committed by GitHub <noreply@github.com>
on 6/1/2026, 9:08:16 AM