Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211)
Fix AutoTokenizer returning TokenizersBackend for DeepSeek-R1-Distill-Qwen models (#48211) The wildcard `deepseek-ai/deepseek-r1-distill-*` in `MODEL_IDS_TO_TOKENIZERS_BACKEND` was too broad: combined with `.lower()` normalization, it matched Qwen-based distills (e.g. `DeepSeek-R1-Distill-Qwen-1.5B`), routing them to `TokenizersBackend` instead of the correct `Qwen2Tokenizer`. Narrowed to `deepseek-r1-distill-llama-*` so only LLaMA-based distills (which genuinely need `TokenizersBackend`) are captured. Qwen-based distills now follow the normal `qwen2` model-type mapping. History: - efe4049614 (Jun 19) introduced MODEL_IDS_TO_TOKENIZERS_BACKEND with the specific entry "deepseek-ai/deepseek-r1-distill-llama-8b" (only the LLaMA-8B variant) - 35bb43e746 (Jul 13) broadened it to "deepseek-ai/deepseek-r1-distill-*", silently capturing all Qwen-based distills too — regression introduced here - e194978ab2 (Jul 15) added the slow test asserting the Qwen variant uses Qwen2Tokenizer, which started failing immediately in CI Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Y
Yih-Dar committed
51edaf85bae776b4ef0db8e8b6949db8dc7a171b
Parent: 6ec0f83
Committed by GitHub <noreply@github.com>
on 8/26/2026, 5:47:53 AM