[Perf] Conditionally restore the MLX buffer memory default (#637)
Fixes the silent 36% single-stream regression that came with #586. Dropping the MLX_MAX_MB_PER_BUFFER default fixed the #585 startup failure, but nobody measured decode afterwards: on Qwen3.6-35B-A3B-4bit (M1 Ultra) main does 58.2 tok/s where the same code with the old default does 91.4. MLX splits each decode step into command buffers by memory touched, and a large MoE step walks about 2 GB of weights, so without the limit raised every step pays the commit overhead many times. I traced it by rebuilding the campaign-era tree in an isolated vLLM 0.26.0 venv (97.8 tok/s), running today's code under 0.26 (57.6, so not a vLLM regression), then flipping the single env var on main (91.0). Both sides of the trade are real, so the default is conditional and single-tier: 2000 when the usable budget (total memory times the effective memory fraction) is at least 90 GiB, and nothing below that, on Ray executors, or above 4096 batched tokens. The #585 shape is pinned two ways: it never gets a default, and a second engine with a disqualifying shape removes the plugin's own earlier default instead of inheriting it. A manual export always wins. Per review, the extrapolated 512 tier is dropped; I can follow up with it once I have serving numbers from a real mid-memory machine. | MLX_MAX_MB_PER_BUFFER | decode tok/s | profiled overhead | |---|---|---| | unset (main today) | 58.2 | 0.99 GB | | 512 (manual export) | 88.0 | 7.6 GB | | 2000 | 91.4 | 13.6 GB | Serve command for all rows: `VLLM_METAL_MEMORY_FRACTION=0.5 vllm serve mlx-community/Qwen3.6-35B-A3B-4bit --max-num-seqs 8`, decode-only tok/s from streamed completions; only 2000 ships as a default. At 8192 batched tokens, 2000 inflates the overhead to 42 GB and startup fails, reproducing #585 exactly. Greedy output is byte-identical across values. Known limits: offline LLM() defaults to 8192 batched tokens so it stays unchanged, and in-process engines (VLLM_ENABLE_V1_MULTIPROCESSING=0) never adopt the default at all since MLX initializes before the config hook runs, verified live, so they get neither the win nor any new risk; the speedup applies through spawned engine processes. --------- Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
C
Chao-Ju Chen committed
41f3ed3f3cfa673ab8a6e1384dd7b1cd9e34ebda
Parent: 790b09f
Committed by GitHub <noreply@github.com>
on 8/25/2026, 2:08:57 AM