SIGN IN SIGN UP

Use `tokenizer.vocab_size()` instead of hardcoding 32000 in convert-pth-to-ggml.py (#142)

There are ways that special tokens or other new tokens could be added to the tokenizer; therefore it's probably best not to assume the vocabulary is only 32000 tokens.
R
Ronsor committed
956dfda8ad8cea7961e22e0384bbc315bf79aed2
Parent: 113e685
Committed by GitHub <noreply@github.com> on 3/15/2023, 7:37:50 PM