SIGN IN SIGN UP

perf(cache): assemble the clean prompt-boundary cache instead of re-prefilling

A path-dependent store needs the cache as it stood at the prompt boundary.
Recurrent state and rotating windows cannot be recovered from post-generation
state, so the store re-prefilled the entire prompt — a second full forward
pass, running after the response was dispatched and holding the scheduler
while the user's next message waited.

Profiled with py-spy on Qwen3.8-27B at 15.4k tokens:
_prefill_for_clean_path_dependent_cache was 40.8% of the engine profile,
about 28 seconds. The user-visible shape was a conversation whose SECOND
message cost ~36s despite being a full cache hit, while the third cost 1.8s.
A delay sweep pinned it exactly: pausing 0/10/30/60s before message two gave
39.2s / 30.4s / 10.8s / 1.14s.

For the pure-hybrid layout that pass is unnecessary, because both halves of
the boundary state are already in hand. Attention layers are append-only
KVCache, so slicing to the boundary recovers exactly the prompt-boundary KV —
the same operation the truncation path already performs. Recurrent layers were
deep-copied at the boundary during the live prefill by the vmlx#109 inline
capture, before generation advanced them. Assembling those two is exact, not
an approximation.

It stays off wherever slicing would be wrong: ZAYA CCA and mixed-SWA rotating
layouts keep re-prefilling, and quantized attention is refused. Whenever
anything fails to line up the helper returns None and the previous path runs,
so it never fabricates state — it either holds the exact boundary cache or it
declines.

The snapshot is handed over by request_id because the companion store consumes
and clears the inline checkpoints first, and the scheduler holds a different
request wrapper than the generator. That map is bounded so a dropped request
cannot leak.

Measured second-message TTFT: 36.9s -> 2.36s (2.36 / 2.36 / 2.31), a 15.6x
improvement, with fidelity unchanged: partial-hit battery passes, a six-way
divergent suffix matrix answers correctly, reasoning-effort changes and
out-of-set efforts are correct, thinking on/off is correct, and restart
survival still restores 1885 tokens from L2. dots3 mixed-SWA was checked to
confirm the gate stays inert there: zero assemblies, partial hits still pass,
no invariant violations.

Four hypotheses were measured and discarded before this one: store cost
(1.4s), lock contention (metadata only), chunked prefill (one-shot measured
identical), and SSM re-derive (inline capture already avoids it).
J
Jinho Jang committed
f5b82284ea3b2734b0b4ddcbefa7aa66244fcc8d
Parent: 200e6ba