SIGN IN SIGN UP

Stop pairing two-word pretokenized documents in the python tokenizer (#48279)

* Stop pairing two-word pretokenized documents in the python tokenizer

When batching with is_split_into_words=True, PythonBackend treated any
inner list of exactly two strings as a sentence pair and pair-encoded
it, inserting a separator between the two words. The fast backend
encodes the same input as one pretokenized document, so slow and fast
disagreed silently and every two-word document in a batch was
corrupted. Under split mode an inner list is now always a single
pretokenized sequence; tuples keep their existing auto-pair role,
matching the fast backend where a tuple entry means (sequence, pair),
and non-split inputs are unchanged.

Test Plan:
  python -m pytest tests/tokenization/test_tokenization_utils.py -q --no-header -p no:cacheprovider
  fail-before (guard reverted): test_batch_pretokenized_documents_of_two_words_are_not_pairs
  failed with a doubled [SEP] row; pass-after: 23 passed, 1 skipped
  python -m pytest tests/models/ctrl/test_tokenization_ctrl.py tests/models/bertweet/test_tokenization_bertweet.py tests/models/siglip/test_tokenization_siglip.py tests/models/bert_generation/test_tokenization_bert_generation.py -k pretokenized -q --no-header -p no:cacheprovider
  all passed (these feed tuple pairs through the shared mixin and fail under an earlier draft that un-paired tuples)

* Use a real slow tokenizer for the pretokenized batching tests

Swap the hand-written legacy BERT vocab files for
ProphetNetTokenizer.from_pretrained per review, so the regression tests
run against a shipped sentencepiece-based python tokenizer instead of a
synthetic one.
V
Vaggelis Giannopoulos committed
6ec0f83563f56335d4227354614459a01d76ba0d
Parent: 2c1376c
Committed by GitHub <noreply@github.com> on 8/25/2026, 9:04:34 PM