COMMITS
/ tests/test-tokenizer-1-bpe.cpp October 10, 2024
D
common : use common_ prefix for common library functions (#9805)
Diego Devesa committed
July 5, 2024
J
Detokenizer fixes (#8039)
jaime-m-p committed
May 11, 2024
H
llama : lookup word in vocab before doing BPE merges (#7193)
Haoxiang Fei committed
March 11, 2024
G
llama : refactor unicode stuff (#5992)
Georgi Gerganov committed
February 16, 2024
B
ggml : add numa options (#5377)
bmwl committed
February 13, 2024
G
tests : multi-thread the tokenizer tests (#5474)
Georgi Gerganov committed
October 24, 2023
G
Add more tokenizer tests (#3742)
Galunid committed
October 3, 2023
G
Work on the BPE tokenizer (#3252)
goerch committed