enhance: Optimize dropped segment index GC (#49728)
Related to #49585 ## What changed This PR refactors DataCoord GC scheduling and moves dropped segment index cleanup into the dropped segment recycle path. - Split the previous serial `meta` GC loop into independent GC tasks so dropped segment recycling no longer blocks index, segment-index, analyze, JSON/text stats, snapshot, orphan, or LOB cleanup work. - Broadcast GC pause commands to all pausable GC tasks instead of only the old `meta` worker. - Recycle dropped segments concurrently, while preserving the per-segment cleanup order. - When recycling a dropped segment, delete segment object files and segment index files first, then remove segment-index meta, and finally remove segment meta. - Keep orphan index file/meta GC as a fallback for old data and partial failure recovery. - Add `GetAllSegmentIndexes` so dropped segment GC can clean segment-index meta even when the field index has already been marked deleted and would be filtered by `GetSegmentIndexes`. ## Failure and safety behavior - If object file deletion fails, segment-index meta and segment meta are kept for retry. - If segment-index meta deletion fails, segment meta is kept for retry. - Snapshot protection is checked for both segment IDs and index build IDs before deleting files or metadata. - V3 segment data still uses `RemoveWithPrefix` for the manifest base path, and index files are removed from recorded segment-index file keys. ## Tests Added coverage for: - GC task registry and pause worker fan-out. - Dropped segment concurrent worker signaling. - Dropped segment cleanup of binlog plus segment-index files/meta. - Deleted field-index filtering case, where `GetSegmentIndexes` returns empty but dropped segment GC still finds segment-index meta. - File deletion failure, segment-index meta deletion failure, and segment meta deletion failure retry behavior. - Snapshot-blocked index build IDs. - V3 dropped segment file deletion success and failure paths. - `GetAllSegmentIndexes` nil/clone behavior. Local verification: - `gofmt` - `git diff --check` I also attempted targeted `go test -tags dynamic,test ./internal/datacoord ... -coverprofile=...`, but this local workspace has a stale C++ core output mismatch and fails at link time with `undefined reference to loon_properties_inject_external_spec`. CI should run against a matching core build. --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C
cai.zhang committed
b6a7af098e3b2251e397188d1bf4d9cd05310ac8
Parent: 8275d10
Committed by GitHub <noreply@github.com>
on 5/27/2026, 10:06:17 PM