Clamp ragged batched speculative acceptance to fix phantom KV (#1962) (#2113)
Zeroing a rejected row's KV tail during batched speculative rollback left those keys attended (q dot 0 still scores e^0 in the softmax denominator), diluting rows that accepted fewer tokens. Clamp per-row acceptance to a uniform count before rollback, and fail loud if a ragged batch ever reaches a rectangular batch cache.
A
Alazer Manakelew committed
3b8f727dc3895877c9b4b8d8143fcde04d00e2ab
Parent: 0f185b1
Committed by GitHub <noreply@github.com>
on 8/31/2026, 4:39:14 PM