SIGN IN SIGN UP

MAX_SEQS above 12 at CTX=huge kills the engine, and the CG cap does not stop it

Found while trying to reproduce @mjungnickel18's thrashing cell in #25. Same visible
failure as gotcha 38 -- boots, captures graphs, answers /health 200, dies on the first
prompt with torch.OutOfMemoryError -- but a different bill, and the fix for 38 does not
cover it: CG is pinned at its 64 cap in every one of these runs, so the memory is going
to allocations that scale with max_num_seqs itself rather than with the captured batch.

Free VRAM after boot on one 24 GiB 3090, SPEC=dflash2 CTX=huge PREFIX_CACHE=1 k=7, then
a single ~3.7k-token prompt:

  MAX_SEQS=8   596 MiB  ok
  MAX_SEQS=10  456 MiB  ok
  MAX_SEQS=12  416 MiB  ok
  MAX_SEQS=16  356 MiB  DEAD

  expandable_segments: memory mapping failed with OOM on device 0 while trying to
  map 20971520 bytes (free: 20578304, total: 25272516608)

20 MB wanted against 20 MB left on a 24 GB card. It needs no concurrency at all --
num_running_reqs=1, step_counter=0, kv_cache_usage=0.18. Reproduced twice with
byte-identical counters. The prefill's transient set is ~356 MiB at
--max-num-batched-tokens 2048, which is why 12 clears it by ~60 MiB and 16 does not.

Warns rather than clamps. Unlike CG this is a VRAM budget, not a shape, and a card
larger than 24 GiB will have room where this one does not; KV_MEM is the knob that buys
the headroom back. The shipped CTX=huge default is MAX_SEQS=2, so nothing here is
reachable without an override.

bench/seat_ttft.py is the probe: cold-then-warm TTFT at N=1 with a per-request salt, so
"warm" means warm engine rather than warm cache. It reports a contentless stream as a
data point instead of raising -- the same trap as the content-or-empty fix in
bugb_sweep.py, and it is what made this failure visible rather than a TypeError.
M
mhenrichsen committed
67b89453b5b54b36c81024934b00d800f89c0f4b
Parent: 7db2d1f