Add tokenizer test + revert to C++11 (#355)
* Add test-tokenizer-0 to do a few tokenizations - feel free to expand * Added option to convert-pth-to-ggml.py script to dump just the vocabulary * Added ./models/ggml-vocab.bin containing just LLaMA vocab data (used for tests) * Added utility to load vocabulary file from previous point (temporary implementation) * Avoid using std::string_view and drop back to C++11 (hope I didn't break something) * Rename gpt_vocab -> llama_vocab * All CMake binaries go into ./bin/ now
G
Georgi Gerganov committed
eb34620aeceaf9d9df7fcb19acc17ad41b9f60f8
Parent: 2e664f1
Committed by GitHub <noreply@github.com>
on 3/21/2023, 3:29:41 PM