perf(cache): assemble the clean prompt-boundary cache instead of re-prefilling
A path-dependent store needs the cache as it stood at the prompt boundary. Recurrent state and rotating windows cannot be recovered from post-generation state, so the store re-prefilled the entire prompt — a second full forward pass, running after the response was dispatched and holding the scheduler while the user's next message waited. Profiled with py-spy on Qwen3.8-27B at 15.4k tokens: _prefill_for_clean_path_dependent_cache was 40.8% of the engine profile, about 28 seconds. The user-visible shape was a conversation whose SECOND message cost ~36s despite being a full cache hit, while the third cost 1.8s. A delay sweep pinned it exactly: pausing 0/10/30/60s before message two gave 39.2s / 30.4s / 10.8s / 1.14s. For the pure-hybrid layout that pass is unnecessary, because both halves of the boundary state are already in hand. Attention layers are append-only KVCache, so slicing to the boundary recovers exactly the prompt-boundary KV — the same operation the truncation path already performs. Recurrent layers were deep-copied at the boundary during the live prefill by the vmlx#109 inline capture, before generation advanced them. Assembling those two is exact, not an approximation. It stays off wherever slicing would be wrong: ZAYA CCA and mixed-SWA rotating layouts keep re-prefilling, and quantized attention is refused. Whenever anything fails to line up the helper returns None and the previous path runs, so it never fabricates state — it either holds the exact boundary cache or it declines. The snapshot is handed over by request_id because the companion store consumes and clears the inline checkpoints first, and the scheduler holds a different request wrapper than the generator. That map is bounded so a dropped request cannot leak. Measured second-message TTFT: 36.9s -> 2.36s (2.36 / 2.36 / 2.31), a 15.6x improvement, with fidelity unchanged: partial-hit battery passes, a six-way divergent suffix matrix answers correctly, reasoning-effort changes and out-of-set efforts are correct, thinking on/off is correct, and restart survival still restores 1885 tokens from L2. dots3 mixed-SWA was checked to confirm the gate stays inert there: zero assemblies, partial hits still pass, no invariant violations. Four hypotheses were measured and discarded before this one: store cost (1.4s), lock contention (metadata only), chunked prefill (one-shot measured identical), and SSM re-derive (inline capture already avoids it).
J
Jinho Jang committed
f5b82284ea3b2734b0b4ddcbefa7aa66244fcc8d
Parent: 200e6ba