0 0 0 C++

command-r : add BPE pre-tokenization (#7063)

* Add BPE pre-tokenization for Command-R/R+.

* Bump transformers convert requirement.

* command-r : add individual digits regex

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

DAN™ committed 2y ago

889bdd76866ea31a7625ec2dcea63ff469f3e981

Parent: 6fbd432

Committed by GitHub <noreply@github.com> on 5/5/2024, 5:19:30 AM