0 0 0 C++

Respect tokenizer.ggml.add_bos_token value when tokenizing (#4040)

* gguf-py: gguf-dump: Respect --no-tensor flag in JSON mode.

* Respect add_bos_token GGUF metadata value

* gguf-py: Try to fix SpecialVocab giving up too easily for the Nth time

Kerfuffle committed 2y ago

91f6499393d2d999331fbfdba47a7f8b9f913f0d

Parent: 8da4627

Committed by GitHub <noreply@github.com> on 11/17/2023, 2:14:37 AM