Offload the vision tower to host RAM, so VISION=1 works with SPEC=dflash2 (#36)
VISION=1 (#23) works on SPEC=mtp and not on SPEC=dflash2, which is the mode the
README recommends to anyone who owns the card. On this 3090 the dflash2 path dies
in graph capture:
torch.OutOfMemoryError: Tried to allocate 960.00 MiB.
GPU 0 has 23.55 GiB of which 787.50 MiB is free.
spec_decode_attn.py:184, self.part_o = torch.empty(...)
The dflash2 pool is pinned by bytes (KV_MEM), so the 0.858 GiB tower cannot come
out of the KV cache -- it comes out of the ~1.1 GiB transient margin the KV_MEM
comment sizes, and it is 0.85 of it. The split-KV verify buffer then has nowhere
to go. SPEC=mtp has no such problem: its pool is profiling-sized, and what the
tower costs there is buried in the +-0.87 GiB profiling swing gotcha 9 already
warns about (79,271 against 80,055 tokens across a VISION=0/1 pair -- 1%, noise).
vLLM can already put weights in pinned host memory, but its offloader is only
installed in make_layers() (model_executor/models/utils.py), which the language
decoder layers use and the ViT does not -- Qwen3_VisionTransformer builds
self.blocks as a plain nn.ModuleList. So --cpu-offload-gb cannot reach the tower,
and setting it spends the budget on language layers instead, which puts PCIe in
the decode hot loop.
patches/vision-tower-cpu-offload.patch routes the tower's blocks and merger
(870 of its 878.8 MiB) through their own UVAOffloader -- dedicated, so it never
competes with --cpu-offload-gb -- with zero-copy off, leaving the bulk-copy path:
each module is copied H2D once per forward and freed. patch_embed and pos_embed
(8.5 MiB) stay resident, so visual.device and visual.dtype still report cuda.
Isolated tower, RTX 3090 at PCIe 4.0 x16, one 8192-patch image (the
2048-image-token cap the start scripts ship), median of 10 forwards:
VLLM_VISION_CPU_OFFLOAD_GB weights on GPU peak alloc forward
0 (stock) 891.3 MiB 1160.5 MiB 296 ms
1 9.0 MiB 308.2 MiB 333 ms
Output is bit-exact against the resident tower (same state_dict, torch.equal).
Do not use vLLM's UVA zero-copy path for this: same memory, 3327 ms per image,
because the GEMMs then re-read operand tiles over PCIe in the inner loop
(~290 MB/s effective). The patch forces the copy path on its own instance rather
than via VLLM_WEIGHT_OFFLOADING_DISABLE_UVA, so --cpu-offload-gb is unaffected.
VISION_OFFLOAD defaults to 1 in both start scripts, because the alternative is a
config that hard-OOMs, and the cost is 36 ms per image. End to end with
VISION=1 SPEC=dflash2: full 69,758-token pool at CTX=fast, and an OCR round-trip
that reads the pixels.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> D
Daniel Donaldson committed
98e152c4fbb5d9cb394f344389d06a43ae849d63
Parent: 3a68e3b
Committed by GitHub <noreply@github.com>
on 8/25/2026, 12:04:52 PM