0 0 0 C++

Tokenizer WPM fixes (#7500)

* Update random test: add_bos_token.
* Update random test: add WPM models for testing.
* Build vocab.special_tokens_cache using vocab token types.
* Fix and improve WPM preprocessing.
  - Fix unicode edge case combinations.
  - Split by whitspace in the same pass.
* Discard all tokens when no matching found.

jaime-m-p committed 2y ago

02c1ecad07f0e2d2febe8196271bcc64bdc9c006

Parent: 6bd12ce

Committed by GitHub <noreply@github.com> on 5/28/2024, 7:46:34 PM