SIGN IN SIGN UP

perf(mtp): verify prefetch - Qwen 4D 36.5 t/s, +65 percent over baseline

Submit the NEXT verify forward with mx.async_eval the moment a cycle's
decision refills the emit queue, so the GPU crunches it while the host spends
~5ms per token on detokenize/stream/stats for the 2-3 queued tokens. Same
forwards, zero extra compute - pure overlap of the measured 10-18ms/cycle
host gap, the same structure as MTPLX's pipelined _commit_pair loop.

The in-flight verify appends unverified draft positions to the backbone
cache, so abandon hooks roll them back from the stored pre-verify snapshot at
every exit that reuses or persists the cache: request finish (before the
prefix-cache store), AR fallback (before the plain _step), cancellation, and
error. Without those, unverified tokens would leak into the stored prefix.

Live, Qwen3.8-27B-JANG_4D-CRACK, code workload, pure defaults:
  36.5 t/s, reproduced exactly (2714-token output, byte-identical runs)
  accept_by_depth d1=1016/1085 (93.6%), d2=580/635 (91.3%), d3 64.6%
  vs 22.1 t/s baseline at the start of this campaign = +65 percent

Also recorded: MLX 0.32.1 upgrade REJECTED by measurement - it regresses our
verify shapes (seq=2 42.2 -> 49.8ms, -15 percent on the dominant shape) while
improving seq>=4; stay on 0.31.2 until block sizes >= 4 matter. Newer mlx-vlm
collides with our patches (target_verify kwarg - upstream now has verify
support; adoption candidate when DFlash lands).

VMLX_MTP_VERIFY_PREFETCH=0 disables. MTP suite 337 green.
J
Jinho Jang committed
3219cff3d69c4976ed2f05ace5c02b1b9045f4d8
Parent: 21b420b