SIGN IN SIGN UP

Improve handling of special tokens in GGML to GGUF converter (#2725)

* Improve UNK, BOS, EOS token handling when converting without metadata.

* Allow importing as a module.

* Remove some obsolete code and minor cleanups.

* Set default UNK token mapping from -1 to 0 in llama.cpp

* Try to handle overflow due to buggy Windows Python with a better error message
K
Kerfuffle committed
777f42ba18b29f25c71ff8de3ecf97b8017304c0
Parent: 46ef5b5
Committed by GitHub <noreply@github.com> on 8/22/2023, 11:39:39 PM