SIGN IN SIGN UP

The seats that break CTX=huge, and the workload that does not

Two more rows for the concurrency section, both from bench/conc_ladder.py on eight
independent 16k-token streams:

- CTX=huge with the shipped MAX_SEQS=2 against the 8 the old README recommended:
  0 preemptions and 14.4 tok/s end-to-end, against 10 preemptions and 10.3. Admitting
  fewer requests gets 40% more work out of the card, which is the whole argument in one
  measurement.
- The same eight streams sharing ONE document with PREFIX_CACHE=1: end-to-end aggregate
  15.5 / 128.4 / 147.8 / 68.3 at N=1/2/4/8, TTFT 14.6 s / 1.3 / 2.6 / 11.5. One stream
  pays the prefill and the rest hit the cache. 148 tok/s at four readers against 15.9
  when the documents differ is a 10x difference on the same server at the same prompt
  length, and it is the difference between what this mode is for and what it is not.

Units while I was in here: KV_MEM=5261334938 is 4.90 GiB, not the 5.26 the CTX=huge
comment claimed -- 5.26 is the decimal-GB figure. And the state page is quoted as the
measured 15.8% of the pool (~0.82 GiB) with gotcha 33's independent 0.88 GiB fit beside
it, rather than as one number carried between two derivations.
M
mhenrichsen committed
bbc6c0f9351c3e1eaca644006a723de032969585
Parent: 979f83d