:red_circle: Allow tokenizers 0.23.1 (#46381)
* Allow tokenizers 0.23.1 Closes #45736 * Fix RobertaProcessing kwargs for tokenizers 0.23.1 tokenizers 0.23.1 made RobertaProcessing args positional-only. Convert the 5 call sites (convert_slow_tokenizer x2, clip, layoutlmv3, roberta) to positional (sep, cls, trim_offsets, add_prefix_space) — works on 0.22.x and 0.23.1. * Tolerate dropped overflow segment in max-encoding-length tests tokenizers 0.23.1 early-exits right-direction truncation (huggingface/tokenizers#1990), so return_overflowing_tokens may omit the trailing overflow segment when it is shorter than the window. Validate the overflow content only when that segment is present; truncation is always checked. Keeps passing on 0.22.x (segment always present). * Drop overflow-segment assertions in max-encoding-length tests The trailing overflow segment is not essential to validate here, and 0.23.1's truncation early-exit (huggingface/tokenizers#1990) may omit it. Keep the truncation checks; remove the now-dead overflow expectations. * Apply suggestion from @ArthurZucker * Fix HerbertTokenizer: pass BertProcessing sep/cls positionally tokenizers.processors.BertProcessing.__new__ does not accept 'sep' and 'cls' keyword arguments, raising TypeError on tokenizer init. Pass the (token, id) tuples positionally instead. * Update dependency versions table for tokenizers 0.23.1 pin * Remove accidentally committed test_opt tokenizer artifacts * Fix ty type errors in tokenization files Run with tokenizers 0.23.1 (new type stubs). Fixes: - add missing '| None' to params defaulting to None (truncation, udop text/text_target) across layoutlmv2/v3, layoutxlm, luke, markuplm, mluke, roc_bert, pop2piano, udop (invalid-parameter-default) - pop2piano: offset_sec annotated int but defaults to 0.0 -> float - bertweet: import html.entities (submodule not pulled in by 'import html') - tapas: tuple[float, Ellipsis] -> tuple[float, ...] (invalid-type-form) * Pass BertProcessing/RobertaProcessing args positionally in call sites merged from main
A
Arthur committed
242f5df996ded281cc328c5b9bcc1bee0ec48118
Parent: a5f41e0
Committed by GitHub <noreply@github.com>
on 8/17/2026, 1:35:25 AM