CUDA: quantized KV support for FA vec (#7527)
* CUDA: quantized KV support for FA vec * try CI fix * fix commented-out kernel variants * add q8_0 q4_0 tests * fix nwarps > batch size * split fattn compile via extern templates * fix flake8 * fix metal tests * fix cmake * make generate_cu_files.py executable * add autogenerated .cu files * fix AMD * error if type_v != FP16 and not flash_attn * remove obsolete code
J
Johannes Gäßler committed
9b596417af11c9ac44fcae0fcfbc6f3665089083
Parent: a323ec6
Committed by GitHub <noreply@github.com>
on 6/1/2024, 6:44:14 AM