llama : lookup word in vocab before doing BPE merges (#7193)
* fix: llama-3 ignore_merges * test: add test for llama-3 bpe ignore_merges * fix: set ignore_merges only for llama-3 * fix: test-tokenizer-1-bpe --ingore-merges detection * fix: copy to fix fallthrough * fix: change ignore_merges to bool * fix: add ignore merges tests to cmake * llama : alternative merge ignore logic --------- Co-authored-by: Haoxiang Fei <feihaoxiang@idea.edu.cn> Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
H
Haoxiang Fei committed
f99e1e456eaf69cc38c1982a2693ce41c0f897ef
Parent: 5ae3426
Committed by GitHub <noreply@github.com>
on 5/11/2024, 8:12:06 AM