Improve handling of special tokens in GGML to GGUF converter (#2725)
* Improve UNK, BOS, EOS token handling when converting without metadata. * Allow importing as a module. * Remove some obsolete code and minor cleanups. * Set default UNK token mapping from -1 to 0 in llama.cpp * Try to handle overflow due to buggy Windows Python with a better error message
K
Kerfuffle committed
777f42ba18b29f25c71ff8de3ecf97b8017304c0
Parent: 46ef5b5
Committed by GitHub <noreply@github.com>
on 8/22/2023, 11:39:39 PM