SIGN IN SIGN UP

tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)

* tests : add test-tokenizer-0.sh

* unicode : add all unicode number ranges

* starcoder : fix pre-tokenizer

* tests : add test that fails with DeepSeek tokenizers

* falcon : fix regex

* unicode : regenerate unicode tables

* refact : add tokenizer model

* lint : fix

* tests : disable failing tests

ggml-ci

* refact : add tests files

ggml-ci

* convert : print -> logging

ggml-ci

* lint : fix

* unicode : digit -> number

* phi-3 : update
G
Georgi Gerganov committed
92139b90af4841d7fd060b526bdd443b621770ff
Parent: a2ac89d
Committed by GitHub <noreply@github.com> on 5/4/2024, 5:32:32 AM