SIGN IN SIGN UP

Add support for BERT embedding models (#5423)

* BERT model graph construction (build_bert)
* WordPiece tokenizer (llm_tokenize_wpm)
* Add flag for non-causal attention models
* Allow for models that only output embeddings
* Support conversion of BERT models to GGUF
* Based on prior work by @xyzhang626 and @skeskinen

---------

Co-authored-by: Jared Van Bortel <jared@nomic.ai>
Co-authored-by: Jared Van Bortel <cebtenzzre@gmail.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
D
Douglas Hanley committed
2891c8aa9af17f4ff636ff3868bc34ff72b56e25
Parent: 97a3365
Committed by GitHub <noreply@github.com> on 2/11/2024, 4:21:38 PM