SIGN IN SIGN UP

feat(mtp): confidence gate that stops extending a draft chain the head is unsure about

Speculative decoding pays for a draft whether or not it is accepted. On
high-entropy positions the head is guessing, and a deep chain spends forwards
on tokens that are about to be rejected. Measured on Qwen3.8-27B at depth 3:
the same code prompt runs 51.5 t/s with thinking off and 28.1 with thinking
on. That gap is entropy, not the engine, and depth adaptation barely touches
it -- forcing depth 1 on the same reasoning workload only reaches 30.3.

Reads the head's own top-1-minus-top-2 logit gap after the FIRST draft and
stops the chain there when it is small. Once per cycle, not per level: the
branch needs a host sync, so one sync to avoid up to two further head forwards
and a wider verify is the trade that pays, while a sync per level is not.

The gap comes from logits already computed and uses a k=2 partial reduction
rather than a sort -- the sampler's full-vocabulary sort was measured at
roughly 30% of decode on wide-vocabulary bundles, so this deliberately avoids
that shape.

Never drops below one draft, so no caller has to handle an empty chain.

Ships INERT: the threshold defaults to 0. A gate calibrated on one family
would cost a sync per cycle and buy nothing on a model whose head is confident
everywhere, so it turns on per family, by measurement. Cycles where it fires
are counted and reported in the per-request MTP line so a threshold that never
fires, or fires always, is visible instead of inferred from throughput.
J
Jinho Jang committed
0655d3341f7ee9fd7dfa13c3f73111c0be49f098
Parent: d54add1