Stop pairing two-word pretokenized documents in the python tokenizer (#48279)
* Stop pairing two-word pretokenized documents in the python tokenizer When batching with is_split_into_words=True, PythonBackend treated any inner list of exactly two strings as a sentence pair and pair-encoded it, inserting a separator between the two words. The fast backend encodes the same input as one pretokenized document, so slow and fast disagreed silently and every two-word document in a batch was corrupted. Under split mode an inner list is now always a single pretokenized sequence; tuples keep their existing auto-pair role, matching the fast backend where a tuple entry means (sequence, pair), and non-split inputs are unchanged. Test Plan: python -m pytest tests/tokenization/test_tokenization_utils.py -q --no-header -p no:cacheprovider fail-before (guard reverted): test_batch_pretokenized_documents_of_two_words_are_not_pairs failed with a doubled [SEP] row; pass-after: 23 passed, 1 skipped python -m pytest tests/models/ctrl/test_tokenization_ctrl.py tests/models/bertweet/test_tokenization_bertweet.py tests/models/siglip/test_tokenization_siglip.py tests/models/bert_generation/test_tokenization_bert_generation.py -k pretokenized -q --no-header -p no:cacheprovider all passed (these feed tuple pairs through the shared mixin and fail under an earlier draft that un-paired tuples) * Use a real slow tokenizer for the pretokenized batching tests Swap the hand-written legacy BERT vocab files for ProphetNetTokenizer.from_pretrained per review, so the regression tests run against a shipped sentencepiece-based python tokenizer instead of a synthetic one.
V
Vaggelis Giannopoulos committed
6ec0f83563f56335d4227354614459a01d76ba0d
Parent: 2c1376c
Committed by GitHub <noreply@github.com>
on 8/25/2026, 9:04:34 PM