SIGN IN SIGN UP

[Perf] Conditionally restore the MLX buffer memory default (#637)

Fixes the silent 36% single-stream regression that came with #586.
Dropping the MLX_MAX_MB_PER_BUFFER default fixed the #585 startup
failure, but nobody measured decode afterwards: on Qwen3.6-35B-A3B-4bit
(M1 Ultra) main does 58.2 tok/s where the same code with the old default
does 91.4. MLX splits each decode step into command buffers by memory
touched, and a large MoE step walks about 2 GB of weights, so without
the limit raised every step pays the commit overhead many times. I
traced it by rebuilding the campaign-era tree in an isolated vLLM 0.26.0
venv (97.8 tok/s), running today's code under 0.26 (57.6, so not a vLLM
regression), then flipping the single env var on main (91.0).

Both sides of the trade are real, so the default is conditional and
single-tier: 2000 when the usable budget (total memory times the
effective memory fraction) is at least 90 GiB, and nothing below that,
on Ray executors, or above 4096 batched tokens. The #585 shape is pinned
two ways: it never gets a default, and a second engine with a
disqualifying shape removes the plugin's own earlier default instead of
inheriting it. A manual export always wins. Per review, the extrapolated
512 tier is dropped; I can follow up with it once I have serving numbers
from a real mid-memory machine.

| MLX_MAX_MB_PER_BUFFER | decode tok/s | profiled overhead |
|---|---|---|
| unset (main today) | 58.2 | 0.99 GB |
| 512 (manual export) | 88.0 | 7.6 GB |
| 2000 | 91.4 | 13.6 GB |

Serve command for all rows: `VLLM_METAL_MEMORY_FRACTION=0.5 vllm serve
mlx-community/Qwen3.6-35B-A3B-4bit --max-num-seqs 8`, decode-only tok/s
from streamed completions; only 2000 ships as a default. At 8192 batched
tokens, 2000 inflates the overhead to 42 GB and startup fails,
reproducing #585 exactly. Greedy output is byte-identical across values.
Known limits: offline LLM() defaults to 8192 batched tokens so it stays
unchanged, and in-process engines (VLLM_ENABLE_V1_MULTIPROCESSING=0)
never adopt the default at all since MLX initializes before the config
hook runs, verified live, so they get neither the win nor any new risk;
the speedup applies through spawned engine processes.

---------

Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
C
Chao-Ju Chen committed
41f3ed3f3cfa673ab8a6e1384dd7b1cd9e34ebda
Parent: 790b09f
Committed by GitHub <noreply@github.com> on 8/25/2026, 2:08:57 AM