SIGN IN SIGN UP

Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211)

Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211)

The wildcard `deepseek-ai/deepseek-r1-distill-*` in `MODEL_IDS_TO_TOKENIZERS_BACKEND`
was too broad: combined with `.lower()` normalization, it matched Qwen-based distills
(e.g. `DeepSeek-R1-Distill-Qwen-1.5B`), routing them to `TokenizersBackend` instead
of the correct `Qwen2Tokenizer`.

Narrowed to `deepseek-r1-distill-llama-*` so only LLaMA-based distills (which
genuinely need `TokenizersBackend`) are captured. Qwen-based distills now follow
the normal `qwen2` model-type mapping.

History:
- efe4049614 (Jun 19) introduced MODEL_IDS_TO_TOKENIZERS_BACKEND with the specific
  entry "deepseek-ai/deepseek-r1-distill-llama-8b" (only the LLaMA-8B variant)
- 35bb43e746 (Jul 13) broadened it to "deepseek-ai/deepseek-r1-distill-*", silently
  capturing all Qwen-based distills too — regression introduced here
- e194978ab2 (Jul 15) added the slow test asserting the Qwen variant uses Qwen2Tokenizer,
  which started failing immediately in CI

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Y
Yih-Dar committed
51edaf85bae776b4ef0db8e8b6949db8dc7a171b
Parent: 6ec0f83
Committed by GitHub <noreply@github.com> on 8/26/2026, 5:47:53 AM