SIGN IN SIGN UP

conc_ladder: measure the steady state, the pool and the preemptions

Written for #16 and reused to answer #25, where it could not have settled anything as
it stood: it reported one throughput number that mixed prefill with decode, its
"~4k-token prompts" were 11.4k-token ones (44 tokens per filler repetition, not the
~15 the code assumed), and it showed nothing about why a rate fell.

Now:
  decode agg  the server's own generated-token counter over a window that opens when
              the LAST stream gets its first token and closes when the FIRST one
              finishes, so it is the interval in which all N are actually decoding.
              Rows where no such interval exists -- the pool could not hold N -- are
              marked, which is itself the finding.
  ms/pass     wall time per forward pass over that window. Flat ms/pass with rising
              tok/pass is healthy batching; ms/pass rising with N means the step is
              being paid per request.
  preempt     preemptions in the window: a capacity problem, not a speculator one.
  kv%         peak pool occupancy, sampled at 4 Hz.
  spread      per-stream min/median/max, which separates "evenly slow" from "one
              served, the rest starved".

Prompts are salted per stream, per repetition and per N so nothing is ever served out
of the prefix cache, --ctx-tokens sets their length and --n 1,2,4,8 skips the ladder
steps that only cost prefill. And a warning the docstring now carries: once prompts
are long, per-stream tok/s is mostly chunked prefill -- 15 / 17 / 107 across three
streams while ms/pass barely moves.
M
mhenrichsen committed
a3c13f766972d34c848acde0bc103b7f84f1206b
Parent: 7cdca1e