SIGN IN SIGN UP
milvus-io / milvus UNCLAIMED

Milvus is a high-performance, cloud-native vector database built for scalable vector ANN search

0 0 164 Go

feat: add commit_timestamp to SegmentInfo for correct MVCC/TTL/GC on import/CDC segments (#48472)

## What this PR does

Adds `commit_timestamp` (uint64) to `SegmentInfo` and `SegmentLoadInfo`
so that import/CDC segments use their logical commit time — not raw row
timestamps — for all temporal decisions.

**Problem:** When bulk-insert imports rows, the row timestamps generated
during the import process may be slightly older than the actual commit
time. Every time-based check in the system (MVCC snapshot visibility, GC
eligibility, collection-level TTL compaction triggering, delete-buffer
anchoring) sees these outdated timestamps instead of the actual commit
time, producing correctness bugs.

**Solution:**

Two-layer approach:

1. **C++ segcore (load-time overwrite):** During `LoadFieldData`, the
in-memory timestamp column is overwritten to `commit_ts_` when
`commit_ts_ != 0`. This makes existing MVCC/delete logic work correctly
with zero hot-path changes.

2. **Compaction normalization:** During compaction, all compaction paths
(mix/sort/clustering) rewrite row timestamps to `commit_ts` in output
binlogs, then set `CommitTimestamp = 0` on the output segment. After
first compaction, the segment becomes a normal segment with no special
handling needed. `commit_ts` is a temporary state that only exists
between import and first compaction.

**Key design decisions:**

- **TTL field (per-row):** Not affected by commit_ts. TTL field values
represent user-specified expiration intent and are honored as-is.
- **Collection-level TTL:** Uses `max(row_ts, commit_ts)` as the
effective row age to prevent premature expiration.
- **Delete/Upsert before commit_ts:** A delete/upsert with `ts <
commit_ts` does NOT take effect, because the row did not exist at that
time. The original `search_pk(pk, delete_ts)` logic handles this
correctly since `row_ts` is overwritten to `commit_ts`.
- **CommitTimestamp assignment:** Currently TODO in `import_checker.go`,
pending companion PR for 2PC commit flow.

## Changes

- `pkg/proto/data_coord.proto`, `query_coord.proto`, `segcore.proto`:
add `commit_timestamp` field
- `internal/datacoord/segment_info.go`:
`segmentEffectiveTs`/`segmentEffectiveDmlTs`/`effectiveTimestamp`
helpers
- `internal/datacoord/meta.go`: compaction completion mutations set
`CommitTimestamp=0` and normalize position timestamps via
`normalizePositionTimestamp` helper
- `internal/datacoord/compaction_trigger.go`: use `effectiveTimestamp`
for TTL compaction trigger
- `internal/datacoord/handler.go`, `garbage_collector.go`,
`compaction_task_l0.go`: use effective timestamp helpers
- `internal/querycoordv2`: propagate `CommitTimestamp` in
`PackSegmentLoadInfo`
- `internal/querynodev2/delegator`: use `segmentEffectiveTs` for
delete-buffer pin/list anchoring
- `internal/core/src/segcore/ChunkedSegmentSealedImpl`: overwrite
timestamp column at load time when `commit_ts_ != 0`
- `internal/datanode/compactor/timestamp_overwrite.go`: Record/Reader
wrappers for compaction timestamp normalization
-
`internal/datanode/compactor/{mix,merge_sort,sort,clustering}_compactor.go`:
apply timestamp overwrite during compaction

## Tests

- Unit tests: DataCoord helpers, `UpdateCommitTimestamp` operator,
`GenSnapshot` filter, GC eligibility, TTL compaction trigger
- Unit tests: QueryNode `segmentEffectiveTs`, delete-buffer pin at
commit_ts
- Unit tests: `timestamp_overwrite.go` Record/Reader wrappers
- C++ tests: MVCC gate, TTL gate, pre-commit delete not applied, normal
segment unchanged
- Integration tests: MVCC visibility, delete/upsert on import segment,
delete/upsert before commit_ts (should not apply), compaction normalizes
commit_ts to 0

## issue

issue: #48471

design doc:
https://github.com/milvus-io/milvus-design-docs/blob/main/design_docs/20260324-commit-timestamp.md

---------

Signed-off-by: Yihao Dai <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Y
yihao.dai committed
0ca60c5b47c04addb13be7053dd8750a50c58949
Parent: e29025b
Committed by GitHub <noreply@github.com> on 5/26/2026, 2:50:33 AM