Support remote GGUF references (#635)
This PR is: - To add remote GGUF serving for `repo_id:quant` references. - To extend the existing local-only GGUF path without changing local GGUF behavior. - To support local or Hugging Face tokenizer/config sources for GGUF weights. - To fail clearly for unsupported qtypes, sharded GGUF, ambiguous matches, and multimodal GGUF. Before this PR, vllm-metal only supported local `.gguf` weight files. After this PR, users can serve either a local GGUF file or a remote GGUF reference: ```bash vllm serve Qwen/Qwen3-0.6B-GGUF:Q8_0 \ --tokenizer Qwen/Qwen3-0.6B ``` ```bash vllm serve /local/model.gguf \ --tokenizer Qwen/Qwen3-0.6B ``` Remote GGUF support is intentionally limited to single-file `Q8_0`, `Q4_0`, and `Q4_1` checkpoints for now. --------- Signed-off-by: Yuan Lik Xun <lxyuan0420@gmail.com>
L
Lik Xun Yuan (Lx) committed
3c9bbdc8839d6bbe99ad4e5b9ad652b15d25da8e
Parent: acce614
Committed by GitHub <noreply@github.com>
on 8/22/2026, 1:07:07 PM