SIGN IN SIGN UP

llama : add support for Tekken pre-tokenizer (#8579)

* llama : Added support for Tekken pre-tokenizer (#8577)

Removed uneeded `vocab.tokenizer_clean_spaces` assignment

* llama : fix order of pre-tokenizers

* * Tekken pre-tokenizer no longer uses clean_up_tokenization_spaces
* Updated chkhsh for Tekken tokenizer

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
M
Michael Coppola committed
940362224d20e35f13aa5fd34a0d937ae57bdf7d
Parent: 69b9945
Committed by GitHub <noreply@github.com> on 7/20/2024, 1:43:51 PM