MAX_SEQS above 12 at CTX=huge kills the engine, and the CG cap does not stop it
Found while trying to reproduce @mjungnickel18's thrashing cell in #25. Same visible failure as gotcha 38 -- boots, captures graphs, answers /health 200, dies on the first prompt with torch.OutOfMemoryError -- but a different bill, and the fix for 38 does not cover it: CG is pinned at its 64 cap in every one of these runs, so the memory is going to allocations that scale with max_num_seqs itself rather than with the captured batch. Free VRAM after boot on one 24 GiB 3090, SPEC=dflash2 CTX=huge PREFIX_CACHE=1 k=7, then a single ~3.7k-token prompt: MAX_SEQS=8 596 MiB ok MAX_SEQS=10 456 MiB ok MAX_SEQS=12 416 MiB ok MAX_SEQS=16 356 MiB DEAD expandable_segments: memory mapping failed with OOM on device 0 while trying to map 20971520 bytes (free: 20578304, total: 25272516608) 20 MB wanted against 20 MB left on a 24 GB card. It needs no concurrency at all -- num_running_reqs=1, step_counter=0, kv_cache_usage=0.18. Reproduced twice with byte-identical counters. The prefill's transient set is ~356 MiB at --max-num-batched-tokens 2048, which is why 12 clears it by ~60 MiB and 16 does not. Warns rather than clamps. Unlike CG this is a VRAM budget, not a shape, and a card larger than 24 GiB will have room where this one does not; KV_MEM is the knob that buys the headroom back. The shipped CTX=huge default is MAX_SEQS=2, so nothing here is reachable without an override. bench/seat_ttft.py is the probe: cold-then-warm TTFT at N=1 with a per-request salt, so "warm" means warm engine rather than warm cache. It reports a contentless stream as a data point instead of raising -- the same trap as the content-or-empty fix in bugb_sweep.py, and it is what made this failure visible rather than a TypeError.
M
mhenrichsen committed
67b89453b5b54b36c81024934b00d800f89c0f4b
Parent: 7db2d1f