Bump vLLM to 0.28.0 (#645)
0.28.0 refactored the KV cache page sizing (vllm#51704) and our TurboQuant spec stopped working: real_page_size_bytes is just an alias now, so the scheduler went back to budgeting blocks with the dense int8 formula. I moved the packed page size into the state_content_bytes field, which is how upstream's own TurboQuant backend does it, and made the hybrid realign update mamba_block_size as well, otherwise the coordinator asserts at startup. The other two changes are new defaults. 0.28.0 turns prefix caching on by default for hybrid models (vllm#50991) and async scheduling on for draft-model spec decode (vllm#48341). Both are resolved before our platform hook sees the config, so if we kept raising like before, a plain vllm serve that worked on 0.27.1 would now die. The hook disables them with a warning instead. An explicit --mamba-block-size still fails fast. We also get to delete a shim this time. The UniProc VLLM_HOST_IP workaround can go, since 0.28.0 does rendezvous through a file:// store now (vllm#50999). That is the only one: vllm#48724 is still not in, CoreEngineActorManager still calls bare ray.init(), and vLLM still caps fastapi below 0.137, so those all stay. I also added huggingface_hub>=1.27.0 to pyproject because we import it directly and vLLM added the floor this release. For evidence, MetalPlatform resolves after a clean ./install.sh, 1874 non-slow tests pass, both serve smokes return their goldens (the Qwen3.5 one runs with the new APC-on default), a TurboQuant hybrid serve boots, and the n-gram e2e passes 7/7. The slow suite has three failures on my machine (the hybrid APC parity check, the draft-model e2e, and a test-ordering memory issue). I reinstalled the 0.27.1 wheel and they fail the same way there, so they were already broken before this bump. `vllm bench serve`, Qwen3-0.6B, sonnet, 100 prompts, `--request-rate inf --temperature 0 --ignore-eos --seed 0`, each arm on its own install. The first 0.28.0 session came out slow because it was the first run after a fresh install, so I dropped the first session of both arms and kept two warm ones each. All sessions completed 100 requests with 54256 input and 15000 generated tokens. | Metric | vLLM 0.27.1 | vLLM 0.28.0 | Delta | |---|---:|---:|---:| | Output tok/s | 1068.84 | 1070.41 | +0.1% | | Total tok/s | 4934.90 | 4942.15 | +0.1% | | Mean TTFT (ms) | 3752.78 | 3727.07 | -0.7% | | P99 TTFT (ms) | 6848.13 | 6832.76 | -0.2% | | Mean TPOT (ms) | 67.58 | 67.62 | +0.1% | | P99 TPOT (ms) | 85.28 | 85.45 | +0.2% | | Duration (s) | 14.03 | 14.01 | -0.1% | --------- Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
C
Chao-Ju Chen committed
db4d7c57c7d72f731f7010fd3b60fc25fc489229
Parent: 24c9c5b
Committed by GitHub <noreply@github.com>
on 8/26/2026, 1:39:22 PM