fix(benchmarks): repair retain_memory's imports of removed engine symbols (#3789)
The retain memory benchmark could not start: both symbols it imports from the engine had been removed out from under it. - `count_tokens_windowed` was deleted by #3788, which moved the tokenizer to quicktok. Its count-only API allocates nothing and is exact, so the windowed approximation #3756 introduced has no reason to exist; the benchmark now calls `count_tokens`. - `_split_contents_into_sub_batches` was replaced by the streaming `_iter_raw_sub_batches` in #3770. The row now consumes the iterator instead of listing it, which is what the retain loop does. Also drops the comment describing tiktoken's ~80 MB encoding table, which no longer applies. Verified by running the benchmark at --mb 2.
N
Nicolò Boschi committed
c2486a2dc47223ecf4261c5ea12744782682f562
Parent: 9fcb7ca
Committed by GitHub <noreply@github.com>
on 8/25/2026, 1:35:12 PM