SIGN IN SIGN UP

:red_circle: Allow tokenizers 0.23.1 (#46381)

* Allow tokenizers 0.23.1

Closes #45736

* Fix RobertaProcessing kwargs for tokenizers 0.23.1

tokenizers 0.23.1 made RobertaProcessing args positional-only. Convert
the 5 call sites (convert_slow_tokenizer x2, clip, layoutlmv3, roberta)
to positional (sep, cls, trim_offsets, add_prefix_space) — works on
0.22.x and 0.23.1.

* Tolerate dropped overflow segment in max-encoding-length tests

tokenizers 0.23.1 early-exits right-direction truncation
(huggingface/tokenizers#1990), so return_overflowing_tokens may omit the
trailing overflow segment when it is shorter than the window. Validate
the overflow content only when that segment is present; truncation is
always checked. Keeps passing on 0.22.x (segment always present).

* Drop overflow-segment assertions in max-encoding-length tests

The trailing overflow segment is not essential to validate here, and
0.23.1's truncation early-exit (huggingface/tokenizers#1990) may omit it.
Keep the truncation checks; remove the now-dead overflow expectations.

* Apply suggestion from @ArthurZucker

* Fix HerbertTokenizer: pass BertProcessing sep/cls positionally

tokenizers.processors.BertProcessing.__new__ does not accept 'sep' and
'cls' keyword arguments, raising TypeError on tokenizer init. Pass the
(token, id) tuples positionally instead.

* Update dependency versions table for tokenizers 0.23.1 pin

* Remove accidentally committed test_opt tokenizer artifacts

* Fix ty type errors in tokenization files

Run with tokenizers 0.23.1 (new type stubs). Fixes:
- add missing '| None' to params defaulting to None (truncation, udop
  text/text_target) across layoutlmv2/v3, layoutxlm, luke, markuplm,
  mluke, roc_bert, pop2piano, udop (invalid-parameter-default)
- pop2piano: offset_sec annotated int but defaults to 0.0 -> float
- bertweet: import html.entities (submodule not pulled in by 'import html')
- tapas: tuple[float, Ellipsis] -> tuple[float, ...] (invalid-type-form)

* Pass BertProcessing/RobertaProcessing args positionally in call sites merged from main
A
Arthur committed
242f5df996ded281cc328c5b9bcc1bee0ec48118
Parent: a5f41e0
Committed by GitHub <noreply@github.com> on 8/17/2026, 1:35:25 AM