0 0 0 C++

BERT tokenizer fixes (#6498)

Key changes:
* BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS
* Nomic Embed conversion: pad vocab instead of slicing embedding tensor
* llama_tokenize: handle added special tokens like HF does

Jared Van Bortel committed 2y ago

1b67731e184e27a465b8c5476061294a4af668ea

Parent: c4a3a4f

Committed by GitHub <noreply@github.com> on 4/9/2024, 5:44:08 PM