The seats that break CTX=huge, and the workload that does not
Two more rows for the concurrency section, both from bench/conc_ladder.py on eight independent 16k-token streams: - CTX=huge with the shipped MAX_SEQS=2 against the 8 the old README recommended: 0 preemptions and 14.4 tok/s end-to-end, against 10 preemptions and 10.3. Admitting fewer requests gets 40% more work out of the card, which is the whole argument in one measurement. - The same eight streams sharing ONE document with PREFIX_CACHE=1: end-to-end aggregate 15.5 / 128.4 / 147.8 / 68.3 at N=1/2/4/8, TTFT 14.6 s / 1.3 / 2.6 / 11.5. One stream pays the prefill and the rest hit the cache. 148 tok/s at four readers against 15.9 when the documents differ is a 10x difference on the same server at the same prompt length, and it is the difference between what this mode is for and what it is not. Units while I was in here: KV_MEM=5261334938 is 4.90 GiB, not the 5.26 the CTX=huge comment claimed -- 5.26 is the decimal-GB figure. And the state page is quoted as the measured 15.8% of the pool (~0.82 GiB) with gotcha 33's independent 0.88 GiB fit beside it, rather than as one number carried between two derivations.
M
mhenrichsen committed
bbc6c0f9351c3e1eaca644006a723de032969585
Parent: 979f83d