SIGN IN SIGN UP

llama : support models without vocabulary (#5798)

* additional methods to read model and ctx parameters

* vocab size as a part of a model metadata

* models without vocabulary, convert.py part

* models without vocabulary, llama.cpp part

* PR clean up

* converter scrypt fixes

* llama_vocab_type update (renamed the new key)

* pr review fixes

* revert function renaming

* one more NoVocab assert
M
Michael Podvitskiy committed
69ff61397d2b7b550dcdda4a35b35128892408b0
Parent: 044ec4b
Committed by GitHub <noreply@github.com> on 3/14/2024, 4:21:56 PM