clip : support more quantization types (#4846)
Uses ggml functions instead of hardcoded names and adds support to quantize into the modern Q-K variants. This is just the bare minimum to get k-types working - a more refined choice of types would be needed to get best quality on low quantizations. I ran a few tests, it doesn't break anything I could notice and a Q6_K ViT works almost as well as Q8_0 but 3 times the inference speed.
J
John committed
d34633d8db6c2e400355de4862cd699154ecc73f
Parent: 4f56458
Committed by GitHub <noreply@github.com>
on 1/10/2024, 1:37:09 PM