llama : support models without vocabulary (#5798)
* additional methods to read model and ctx parameters * vocab size as a part of a model metadata * models without vocabulary, convert.py part * models without vocabulary, llama.cpp part * PR clean up * converter scrypt fixes * llama_vocab_type update (renamed the new key) * pr review fixes * revert function renaming * one more NoVocab assert
M
Michael Podvitskiy committed
69ff61397d2b7b550dcdda4a35b35128892408b0
Parent: 044ec4b
Committed by GitHub <noreply@github.com>
on 3/14/2024, 4:21:56 PM