SIGN IN SIGN UP

llama : implement Unigram tokenizer needed by T5 and FLAN-T5 model families (#5763)

* llama : add T5 model architecture, tensors and model header parameters

* llama : add implementation of Unigram tokenizer with SentencePiece-like text normalization using precompiled charsmap

---------

Co-authored-by: Stanisław Szymczyk <sszymczy@gmail.com>
F
fairydreaming committed
6fcbf6823553efabe52ed83e3c2a3329aa3387d1
Parent: e6bf007
Committed by GitHub <noreply@github.com> on 6/25/2024, 7:14:35 PM