perf(mtp): verify prefetch - Qwen 4D 36.5 t/s, +65 percent over baseline
Submit the NEXT verify forward with mx.async_eval the moment a cycle's decision refills the emit queue, so the GPU crunches it while the host spends ~5ms per token on detokenize/stream/stats for the 2-3 queued tokens. Same forwards, zero extra compute - pure overlap of the measured 10-18ms/cycle host gap, the same structure as MTPLX's pipelined _commit_pair loop. The in-flight verify appends unverified draft positions to the backbone cache, so abandon hooks roll them back from the stored pre-verify snapshot at every exit that reuses or persists the cache: request finish (before the prefix-cache store), AR fallback (before the plain _step), cancellation, and error. Without those, unverified tokens would leak into the stored prefix. Live, Qwen3.8-27B-JANG_4D-CRACK, code workload, pure defaults: 36.5 t/s, reproduced exactly (2714-token output, byte-identical runs) accept_by_depth d1=1016/1085 (93.6%), d2=580/635 (91.3%), d3 64.6% vs 22.1 t/s baseline at the start of this campaign = +65 percent Also recorded: MLX 0.32.1 upgrade REJECTED by measurement - it regresses our verify shapes (seq=2 42.2 -> 49.8ms, -15 percent on the dominant shape) while improving seq>=4; stay on 0.31.2 until block sizes >= 4 matter. Newer mlx-vlm collides with our patches (target_verify kwarg - upstream now has verify support; adoption candidate when DFlash lands). VMLX_MTP_VERIFY_PREFETCH=0 disables. MTP suite 337 green.
J
Jinho Jang committed
3219cff3d69c4976ed2f05ace5c02b1b9045f4d8
Parent: 21b420b