SIGN IN SIGN UP

Concurrency is pool-shaped, not seat-shaped

The README said this "is a single-user mode by configuration, and the knob is
MAX_SEQS", and that MAX_SEQS=8 lifts it. @mjungnickel18 measured that it does not
(#25). He is right, and the paragraph is gone.

What MAX_SEQS lifts is how many requests are ADMITTED. What decides how many RUN is
the pool: every resident request reserves 1+k recurrent-state slots before it stores
one token of context -- 0.88 GiB at DFlash2 k=7, 1.66 GiB at 15, 0.44 GiB at MTP k=4
-- so CTX=fast holds seven DFlash2 requests with 128-token prompts, five at 4k and two
at 16k, against eight and four for MTP. Measured with bench/conc_ladder.py, distinct
salted prompts, 250 W, CTX=fast, MAX_SEQS=8:

  streams                1      2      4      8
  dflash2 per-stream   137     97     46     33
  dflash2 aggregate    137    225    309     (5 resident)
  dflash2 ms/pass     25.9   32.8   49.1      --
  mtp per-stream       126    103     46     23
  mtp aggregate        124    212    280    383
  mtp ms/pass         24.8   29.8   43.1   62.3

Three claims replaced by measurements:

- "not a property of the block verify" was right and now has the evidence. Aggregate
  decode climbs for both speculators and nothing is preempted anywhere in that table.
  What concurrency costs is latency: ~7 ms of forward pass per resident request under
  DFlash2, ~5 under MTP, and it is not attention over their context -- with 128-token
  prompts the slope is the same (25.2 -> 45.8 ms from one resident to four).

- "MAX_SEQS=8 lifts it" is worse than wrong on CTX=huge, where it is what breaks:
  eight independent 16k-token streams with the seats forced to 8 preempt ten times,
  peak occupancy 99.4%, per-stream 3 / 7 / 72 tok/s. That profile ships MAX_SEQS=2.

- The C1-C8 cohort tables read as a general concurrency claim. They are eight chat
  prompts of 45-300 tokens at a concurrency limit, and they now say so. On long
  independent streams the mode is prefill-bound instead: eight 16k prompts give an
  end-to-end aggregate of 15.8 tok/s against 131 for one stream, with the decode-only
  aggregate at 183 over the same run and SPEC=mtp reading the same 15.0-15.9.

gotcha 33 keeps its finding -- the pool's SIZE barely moves with MAX_SEQS -- and gains
the one it was being read as excluding: a RUNNING request occupies 15.8% of it.

The launcher now prints what the pool holds when MAX_SEQS is above it, and honours an
explicit CUDAGRAPH_MODE on every path. Without the second there was no way to ask a
dflash2 server for PIECEWISE, so the FULL-against-PIECEWISE comparison this repo
quotes could not be re-measured after a75ee4b changed what FULL does.

Also here because the same testing found it: DFLASH_TOKENS=15 MAX_SEQS=8 booted,
answered /health and then died on the first concurrent batch with an engine-side
torch.OutOfMemoryError. CG = MAX_SEQS x (k+1) = 128 against a graph reservation sized
for the 64 that every shipped default produces. The derived CG is capped at 64 now;
gotcha 38 has it.

And the WSL2 pinned-memory note was in the wrong place: SPEC=dflash2 forces the V2
model runner on every CTX profile, not only CTX=huge, so a clean venv aborts at
"RuntimeError: UVA is not available" before loading weights whatever profile you pick.
It is in docs/docker.md as the first WSL item and in the compose header now. The name
is VLLM_WSL2_ENABLE_PIN_MEMORY; VLLM_WSL_PIN_MEMORY is not a vLLM variable, and this
README carried it for 22 minutes on 21.08 (589daae, fixed in 27f51fa).
M
mhenrichsen committed
0b6bd2e5db1fb220c447f9bf5203d7194ebda295
Parent: a3c13f7