Fix EOS for candidate generators (#47931)
* Crop DFlash drafts at the first EOS `_assisted_decoding` commits a whole block before running the stopping criteria, and `EosTokenCriteria` only inspects the last committed token, so an EOS accepted earlier in the block was missed and generation ran past it. `n_matches` counts only the leading run where the target's argmax matches the draft, so any EOS before the final committed token must have come from the draft. Cropping the draft at its first EOS therefore makes that impossible, and leaves EOS last so the existing criteria fire unchanged. This mirrors `PromptLookupCandidateGenerator`, which already crops its drafts for the same reason. The crop is gated on `generation_config._eos_token_tensor`, the same tensor `EosTokenCriteria` is built from, so a block is never shortened for a token that would not have stopped generation. The candidate logits are cropped alongside the tokens, since they are consumed positionally. Fixes #47912. AI-assisted: drafted with Claude Code and reviewed before submitting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Crop MTP drafts at the first EOS `_assisted_decoding` commits a whole block before running the stopping criteria, and `EosTokenCriteria` only inspects the last committed token, so an EOS accepted earlier in the block was missed and generation ran past it. `n_matches` counts only the leading run where the target's argmax matches the draft, so any EOS before the final committed token must have come from the draft. Cropping the draft at its first EOS therefore makes that impossible, and leaves EOS last so the existing criteria fire unchanged. This is the same crop #47931 applies to `DFlashTokenCandidateGenerator`, and the one `PromptLookupCandidateGenerator` has always applied to its drafts. It only fires when the accepted EOS is not the final drafted token, so a drafter with `num_mtp_layers == 1` cannot reach it -- its single draft position is also the last, and the existing guard in `_assisted_decoding` covers that. Released one-layer checkpoints (DeepSeek-V3, GLM-4.5) are therefore unaffected; Inkling ships eight mtp layers and is not. The crop is gated on `generation_config._eos_token_tensor`, the same tensor `EosTokenCriteria` is built from, so a block is never shortened for a token that would not have stopped generation. The candidate logits are cropped alongside the tokens, since they are consumed positionally. The mtp cache is untouched: every layer has already run and recorded its position by then, and the cropping done on the next call is driven by `num_mtp_layers`, never by how many candidates were returned. Part of #47912. AI-assisted: drafted with Claude Code and reviewed before submitting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Align the pinned DFlash test head with #47877 and #48007 The stub only had a forward. The device placement from #47877 reads .weight.device on the output embeddings, and #48007 drops the anchor position before the head rather than after, so the pinned rows now map 1:1 to the draft. Both branches were green in isolation; the merged tree failed every DFlash crop test without this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * no tests made by AI please, gotta add smth ourselves at some point --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: raushan <raushan@huggingface.co>
J
Jordan Anderson committed
5b90fc376e42ddf8c4856ed5114cb066f2a630ed
Parent: f471539
Committed by GitHub <noreply@github.com>
on 8/18/2026, 8:56:44 AM