Implement non-greedy tokenizer that tries to maximize token lengths (#242)
* Implement non-greedy tokenizer that tries to maximize token lengths * Insert single space in front of the prompt - this is to match original llama tokenizer behavior --------- Co-authored-by: Jakub Horak <jakub.horak@ibawizard.net>
T
thement committed
c9f670a17755311aa28c411f5c7f3c8c05434770
Parent: 4f54609
Committed by GitHub <noreply@github.com>
on 3/17/2023, 8:05:58 PM