SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 148 Go

enhance: Optimize dropped segment index GC (#49728)

Related to #49585

## What changed

This PR refactors DataCoord GC scheduling and moves dropped segment
index cleanup into the dropped segment recycle path.

- Split the previous serial `meta` GC loop into independent GC tasks so
dropped segment recycling no longer blocks index, segment-index,
analyze, JSON/text stats, snapshot, orphan, or LOB cleanup work.
- Broadcast GC pause commands to all pausable GC tasks instead of only
the old `meta` worker.
- Recycle dropped segments concurrently, while preserving the
per-segment cleanup order.
- When recycling a dropped segment, delete segment object files and
segment index files first, then remove segment-index meta, and finally
remove segment meta.
- Keep orphan index file/meta GC as a fallback for old data and partial
failure recovery.
- Add `GetAllSegmentIndexes` so dropped segment GC can clean
segment-index meta even when the field index has already been marked
deleted and would be filtered by `GetSegmentIndexes`.

## Failure and safety behavior

- If object file deletion fails, segment-index meta and segment meta are
kept for retry.
- If segment-index meta deletion fails, segment meta is kept for retry.
- Snapshot protection is checked for both segment IDs and index build
IDs before deleting files or metadata.
- V3 segment data still uses `RemoveWithPrefix` for the manifest base
path, and index files are removed from recorded segment-index file keys.

## Tests

Added coverage for:

- GC task registry and pause worker fan-out.
- Dropped segment concurrent worker signaling.
- Dropped segment cleanup of binlog plus segment-index files/meta.
- Deleted field-index filtering case, where `GetSegmentIndexes` returns
empty but dropped segment GC still finds segment-index meta.
- File deletion failure, segment-index meta deletion failure, and
segment meta deletion failure retry behavior.
- Snapshot-blocked index build IDs.
- V3 dropped segment file deletion success and failure paths.
- `GetAllSegmentIndexes` nil/clone behavior.

Local verification:

- `gofmt`
- `git diff --check`

I also attempted targeted `go test -tags dynamic,test
./internal/datacoord ... -coverprofile=...`, but this local workspace
has a stale C++ core output mismatch and fails at link time with
`undefined reference to loon_properties_inject_external_spec`. CI should
run against a matching core build.

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
C
cai.zhang committed
b6a7af098e3b2251e397188d1bf4d9cd05310ac8
Parent: 8275d10
Committed by GitHub <noreply@github.com> on 5/27/2026, 10:06:17 PM