0 0 14 Python

fix(tokenizer): make special token insertion opt-in (#3337)

* fix(tokenizer): make special token insertion opt-in

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* fix(examples): preserve Baichuan special tokens

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

* test(tokenizer): opt in to EOS in formatting coverage

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

---------

Signed-off-by: Alexandros Koumparoulis <akoumparouli@nvidia.com>

Alexandros Koumparoulis committed 1mo ago

c0c1caf749c7035e91b667c2b7807daa03229506

Parent: 70e738b

Committed by GitHub <noreply@github.com> on 8/3/2026, 2:30:30 PM