SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 164 Go

fix: clear import segments NumOfRows when import task retries (#49986)

issue: #49985

## Summary

When a DataNode running an import task fails or returns `Retry`,
DataCoord resets the task to `Pending` and reschedules it on another
node. The preallocated segment IDs are reused on retry, but
`PickSegment` on the DataNode picks **randomly** whenever more than one
segment exists for a `(vchannel, partition)`
(https://github.com/milvus-io/milvus/blob/master/internal/datanode/importv2/util.go#L150-L162),
so the retried attempt can write to a different subset of the
preallocated segments than the failed attempt did.

`NumOfRows` updates from progress reports were monotonic-only and were
never cleared across attempts. As a result, segments touched by the
failed attempt but skipped by the retried attempt kept stale `NumOfRows`
without any insert binlogs being persisted (DataCoord only persists
binlogs on `Completed`,
https://github.com/milvus-io/milvus/blob/master/internal/datacoord/import_task_import.go#L209-L234).
Sort compaction later read this stale `NumOfRows` via
`originSegment.GetNumOfRows()`
(https://github.com/milvus-io/milvus/blob/master/internal/datacoord/import_util.go#L899),
found zero binlog rows for the orphan segment, and failed with
`unexpected row count` or EOF
(https://github.com/milvus-io/milvus/blob/master/internal/datanode/compactor/sort_compaction.go#L335-L343).

## Fix

- New operator `ResetImportingSegmentRows(segmentIDs ...int64)` in
`internal/datacoord/meta.go` — clears `NumOfRows` and `MaxRowNum` only
on segments still in `Importing` state.
- In `(*importTask).QueryTaskOnWorker`, call it on every preallocated
segment **before** transitioning the task back to `Pending`. If the
reset fails, the state transition is skipped so the next polling tick
retries the whole sequence.
- Orphan segments touched by the failed attempt but skipped by the
retried attempt now have `NumOfRows = 0` after the retry completes, so
`createSortCompactionTask`'s zero-rows guard drops them instead of
feeding stale counts into sort compaction.

## Test plan

- [x]
`TestImportTask_QueryTaskOnWorker/QueryImport_rpc_failed_resets_NumOfRows`
— new subtest verifies that `NumOfRows` on every preallocated segment is
reset to 0 and the task transitions to `Pending` when QueryImport
returns an error.
- [x] Existing `TestImportTask_QueryTaskOnWorker` subtests still pass
(the existing `QueryImport rpc failed` subtest's meta was extended with
`segments: NewSegmentsInfo()` so the reset code path runs without
nil-deref).
- [x] `golangci-lint run ./internal/datacoord/` is clean.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Y
yihao.dai committed
650f81d89bbd19f93de8593453e225b5ea37df43
Parent: 1f123a8
Committed by GitHub <noreply@github.com> on 5/25/2026, 5:54:36 PM