SIGN IN SIGN UP

llama : support StableLM 2 1.6B (#5052)

* llama : support StableLM 2 1.6B

* convert : fix Qwen's set_vocab wrongly naming all special tokens [PAD{id}]

* convert : refactor Qwen's set_vocab to use it for StableLM 2 too

* nix : add tiktoken to llama-python-extra

* convert : use presence of tokenizer.json to determine StableLM tokenizer loader

It's a less arbitrary heuristic than the vocab size.
C
compilade committed
d6bd4d46ddb6926087c11e0f6633ab1c81da58c3
Parent: 152d9d0
Committed by GitHub <noreply@github.com> on 1/22/2024, 11:21:52 AM