Concurrency is pool-shaped, not seat-shaped
The README said this "is a single-user mode by configuration, and the knob is MAX_SEQS", and that MAX_SEQS=8 lifts it. @mjungnickel18 measured that it does not (#25). He is right, and the paragraph is gone. What MAX_SEQS lifts is how many requests are ADMITTED. What decides how many RUN is the pool: every resident request reserves 1+k recurrent-state slots before it stores one token of context -- 0.88 GiB at DFlash2 k=7, 1.66 GiB at 15, 0.44 GiB at MTP k=4 -- so CTX=fast holds seven DFlash2 requests with 128-token prompts, five at 4k and two at 16k, against eight and four for MTP. Measured with bench/conc_ladder.py, distinct salted prompts, 250 W, CTX=fast, MAX_SEQS=8: streams 1 2 4 8 dflash2 per-stream 137 97 46 33 dflash2 aggregate 137 225 309 (5 resident) dflash2 ms/pass 25.9 32.8 49.1 -- mtp per-stream 126 103 46 23 mtp aggregate 124 212 280 383 mtp ms/pass 24.8 29.8 43.1 62.3 Three claims replaced by measurements: - "not a property of the block verify" was right and now has the evidence. Aggregate decode climbs for both speculators and nothing is preempted anywhere in that table. What concurrency costs is latency: ~7 ms of forward pass per resident request under DFlash2, ~5 under MTP, and it is not attention over their context -- with 128-token prompts the slope is the same (25.2 -> 45.8 ms from one resident to four). - "MAX_SEQS=8 lifts it" is worse than wrong on CTX=huge, where it is what breaks: eight independent 16k-token streams with the seats forced to 8 preempt ten times, peak occupancy 99.4%, per-stream 3 / 7 / 72 tok/s. That profile ships MAX_SEQS=2. - The C1-C8 cohort tables read as a general concurrency claim. They are eight chat prompts of 45-300 tokens at a concurrency limit, and they now say so. On long independent streams the mode is prefill-bound instead: eight 16k prompts give an end-to-end aggregate of 15.8 tok/s against 131 for one stream, with the decode-only aggregate at 183 over the same run and SPEC=mtp reading the same 15.0-15.9. gotcha 33 keeps its finding -- the pool's SIZE barely moves with MAX_SEQS -- and gains the one it was being read as excluding: a RUNNING request occupies 15.8% of it. The launcher now prints what the pool holds when MAX_SEQS is above it, and honours an explicit CUDAGRAPH_MODE on every path. Without the second there was no way to ask a dflash2 server for PIECEWISE, so the FULL-against-PIECEWISE comparison this repo quotes could not be re-measured after a75ee4b changed what FULL does. Also here because the same testing found it: DFLASH_TOKENS=15 MAX_SEQS=8 booted, answered /health and then died on the first concurrent batch with an engine-side torch.OutOfMemoryError. CG = MAX_SEQS x (k+1) = 128 against a graph reservation sized for the 64 that every shipped default produces. The derived CG is capped at 64 now; gotcha 38 has it. And the WSL2 pinned-memory note was in the wrong place: SPEC=dflash2 forces the V2 model runner on every CTX profile, not only CTX=huge, so a clean venv aborts at "RuntimeError: UVA is not available" before loading weights whatever profile you pick. It is in docs/docker.md as the first WSL item and in the compose header now. The name is VLLM_WSL2_ENABLE_PIN_MEMORY; VLLM_WSL_PIN_MEMORY is not a vLLM variable, and this README carried it for 22 minutes on 21.08 (589daae, fixed in 27f51fa).
M
mhenrichsen committed
0b6bd2e5db1fb220c447f9bf5203d7194ebda295
Parent: a3c13f7