llama : allow raw byte in SPM vocabs; don't crash on nl 404 (#5478)
* common : don't crash if newline token is not found * common : llama_byte_to_token: allow falling back to finding just the token byte in SPM vocabs
A
Aarni Koskela committed
c4e6dd59e45ef7b14f7763fb073b517395dc176c
Parent: 037259b
Committed by GitHub <noreply@github.com>
on 2/13/2024, 4:18:16 PM