The 3.5x for PIECEWISE was the residue bug, not the capture mode
This README compared FULL_AND_PIECEWISE against PIECEWISE on the 25k copy task and
reported 38 tok/s (1.97 per step) against 132, "3.5x on the long shared prefix this
mode exists for". Those numbers predate a75ee4b: the FULL column is a corrupted run,
so the table was measuring the bug.
Re-measured at HEAD, same server, only the capture toggled -- which needed the
CUDAGRAPH_MODE branch from the previous commit, because there was no way to ask a
dflash2 server for PIECEWISE any more:
bench/labd_bench.py --ctx 20000, SPEC=dflash2 CTX=huge PREFIX_CACHE=1, decode tok/s
copy code edit quote summary qa all six
FULL 167.1 111.1 84.7 55.0 47.8 43.4 65.7 (3.03/step)
PIECEWISE 166.3 111.3 83.0 62.4 48.6 43.1 67.6 (3.18/step)
The same, five of six within 2%. `quote` is 13% apart, which is greedy divergence on
the task that diverges most and is what puts PIECEWISE 3% ahead overall. At this
context length the capture mode is not a speed decision in either direction, so the
paragraph now says that instead, and the short-prompt 13-18% keeps its upper-bound
marking with @mjungnickel18's 0.2-2.3% beside it. M
mhenrichsen committed
979f83de2d2b165873b1ae095fae6fc7b8f62f5b
Parent: 0b6bd2e