SIGN IN SIGN UP

perf(dots3): bound the prefill lazy graph to eval_every layers

The always-on slot census caught the real dots3 deep-context wall:
cache slots FLAT at 1.1GB (latent KV exactly as designed) while active
Metal sawtoothed 99->118GB chunk to chunk at a 64k context - the lazy
forward graph holds every layer's DSA score buffers (~1GB per full
layer) at once until something evaluates, and the admission valve then
declines against those transient peaks (turn 5 of the 100k ladder
rejected at 64,137 tokens with active=118.28GB vs the 120GB limit).

Materialize the residual stream every 8 layers (env
VMLX_DOTS3_PREFILL_LAYER_EVAL_EVERY, 0 disables) during prefill only:
each segment's score buffers free before the next builds, bounding the
in-flight transient to ~8 layers' worth. Decode is untouched; mx.eval
is semantically neutral so numerics are identical.
J
Jinho Jang committed
b6d14d4549cfed8a6ace6a1bc0799779f3700ce7
Parent: 2422487