SIGN IN SIGN UP

fix(benchmarks): repair retain_memory's imports of removed engine symbols (#3789)

The retain memory benchmark could not start: both symbols it imports from the
engine had been removed out from under it.

- `count_tokens_windowed` was deleted by #3788, which moved the tokenizer to
  quicktok. Its count-only API allocates nothing and is exact, so the windowed
  approximation #3756 introduced has no reason to exist; the benchmark now
  calls `count_tokens`.
- `_split_contents_into_sub_batches` was replaced by the streaming
  `_iter_raw_sub_batches` in #3770. The row now consumes the iterator instead
  of listing it, which is what the retain loop does.

Also drops the comment describing tiktoken's ~80 MB encoding table, which no
longer applies.

Verified by running the benchmark at --mb 2.
N
Nicolò Boschi committed
c2486a2dc47223ecf4261c5ea12744782682f562
Parent: 9fcb7ca
Committed by GitHub <noreply@github.com> on 8/25/2026, 1:35:12 PM