SIGN IN SIGN UP

The 3.5x for PIECEWISE was the residue bug, not the capture mode

This README compared FULL_AND_PIECEWISE against PIECEWISE on the 25k copy task and
reported 38 tok/s (1.97 per step) against 132, "3.5x on the long shared prefix this
mode exists for". Those numbers predate a75ee4b: the FULL column is a corrupted run,
so the table was measuring the bug.

Re-measured at HEAD, same server, only the capture toggled -- which needed the
CUDAGRAPH_MODE branch from the previous commit, because there was no way to ask a
dflash2 server for PIECEWISE any more:

  bench/labd_bench.py --ctx 20000, SPEC=dflash2 CTX=huge PREFIX_CACHE=1, decode tok/s
                copy  code  edit  quote  summary   qa   all six
  FULL         167.1 111.1  84.7   55.0     47.8  43.4  65.7 (3.03/step)
  PIECEWISE    166.3 111.3  83.0   62.4     48.6  43.1  67.6 (3.18/step)

The same, five of six within 2%. `quote` is 13% apart, which is greedy divergence on
the task that diverges most and is what puts PIECEWISE 3% ahead overall. At this
context length the capture mode is not a speed decision in either direction, so the
paragraph now says that instead, and the short-prompt 13-18% keeps its upper-bound
marking with @mjungnickel18's 0.2-2.3% beside it.
M
mhenrichsen committed
979f83de2d2b165873b1ae095fae6fc7b8f62f5b
Parent: 0b6bd2e