0 0 0 C++

Per token attributes (#7685)

* Add per token attributes enum
* Using phi-3 for testing 'rstrip'
* Using jina-v2 for testing 'lstrip'
* Brute force test for 'lstrip' and 'rstrip'
* Implement 'rstrip' and 'lstrip'
* Update phi-3 GGUF file (obsolete since 917dc8c)
* Replace llama_token_type with llama_token_attribs

jaime-m-p committed 2y ago

3b38d48609280aa5f8ab7ea135a4351b2a5ee240

Parent: 6d16169

Committed by GitHub <noreply@github.com> on 6/4/2024, 7:17:17 AM