feat(run-store): make the run-ops router correct at N shards (#4771)
## What
Makes `RoutingRunStore` correct when the run-ops layer routes across
more than two Postgres stores. Today it routes between a gen-1 `new`
dedicated database and a `legacy` control-plane database; this
generalizes every routing policy to N shards while keeping the two-store
behaviour byte-identical.
The change sets the four routing decisions that were implicit in code
order, and fixes one hazard that failed silently:
- **Id → shard key.** The router resolves a shard key with
`resolveShard` instead of the binary residency classifier, so a gen-2 id
reaches its own shard through the keyed map.
- **Membership vs routing.** `#distinctStores` (one entry per physical
database, aliases excluded by a declared `aliasOf`) drives every sum,
probe, and merge; `#shards` drives routing. An aliased shard can no
longer make a sum count one database twice.
- **Probe order.** A keyless lookup stays a sequential short-circuit at
two stores; above two it fans out in parallel, picks by precedence,
tolerates a single down leg, and keeps the canonical not-found throw on
the legacy leg.
- **Precedence and duplicates.** One merge helper across all four merge
sites. A duplicate id confined to `{new, legacy}` stays silent (the
known drain-mirror case); any other cross-shard duplicate increments
`runops_shard_duplicate_id_total` and logs at error level.
- **Disjoint sum (the silent hazard).** `countPendingWaitpoints` and the
waitpoint collector now partition absent ids by shard and **union by
id** rather than summing counts. A drain-mirrored waitpoint on both
gen-1 stores is counted once, so a blocked run can no longer hang
forever on a double-counted pending waitpoint.
- **Waitpoint completion.** A gen-2 waitpoint completes on its own
shard, overriding the legacy pins; a cuid waitpoint keeps its two-member
gen-1-pair probe unchanged.
- **Fail-loud creates.** A create with no shard key throws instead of
silently defaulting to `new`. An id resolving to an unconfigured shard
throws instead of being dropped.
Two new counters are exported: `runops_shard_duplicate_id_total` and
`runops_waitpoint_probe_fallback_total`.
## Why it is safe to merge
With only `{new, legacy}` configured every generalized rule reduces to
today's behaviour. `resolveShard` returns exactly what the old
classifier returned for every id shape that exists today, and no gen-2
id is minted yet. The only intentional behaviour change is the fail-loud
create throw; an enumeration of production call sites confirmed no
caller trips it.
## Testing
- New container-free algebra suite (50 cases) over probe order,
precedence, the duplicate alarm, the disjoint-sum partition, the
waitpoint probes, and the fail-loud paths.
- New `runOpsStore.nShardMatrix.test.ts` runs a four-store matrix
(legacy + new + two gen-2 shards) against real Postgres containers: the
disjoint-sum union, the alias topology, cross-tree completion,
pagination merges, and mixed-id hydration.
- New `makeNShardRunOpsPostgresTest(k)` fixture in
`@internal/testcontainers`.
- Full run-store corpus green: 71 files, 480 tests. Typecheck, lint,
format, and knip all clean.
## Notes
- Draft: opened for review; not marking ready yet.
- No changeset or `.server-changes` file: internal routing
infrastructure, no user-visible behaviour change.
- TRI-13427. D
Daniel Sutton committed
97d70b890696929c1afd1e7700d0f7e2ff0503fc
Parent: ee29393
Committed by GitHub <noreply@github.com>
on 8/25/2026, 2:30:14 PM