llama : cache llama_token_to_piece (#7587)
* llama : cache llama_token_to_piece ggml-ci * llama : use vectors and avoid has_cache ggml-ci * llama : throw on unknown tokenizer types ggml-ci * llama : print a log of the total cache size
G
Georgi Gerganov committed
5921b8f089d3b7bda86aac5a66825df6a6c10603
Parent: 5dcdf94
Committed by GitHub <noreply@github.com>
on 5/30/2024, 4:01:41 PM