Finds llama.cpp-compatible GGUF models on the Hugging Face Hub, picks the right quantization, and runs them locally with llama-cli or an OpenAI-compatible llama-server on CPU, Metal, CUDA, or ROCm.
---
name: huggingface-local-models
description: Select and run models locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUF repos, quant selection, exact file lookup, conversion, and OpenAI-compatible local serving.
---
# Hugging Face Local Models
Search the Hub for llama.cpp-compatible GGUF repos, choose a quant, and launch with `llama-cli` or `llama-server`.
## Workflow
1. Install llama.cpp (`brew install llama.cpp`, winget, or build from source).
2. Search the Hub with `apps=llama.cpp` (e.g. `https://huggingface.co/models?apps=llama.cpp&sort=trending`); prefer the repo's own local-app snippet and quant recommendation.
3. Confirm exact `.gguf` filenames via `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`.
4. Run directly: `llama-cli -hf <repo>:<QUANT>` or `llama-server -hf <repo>:<QUANT>`; fall back to `--hf-repo` + `--hf-file` for custom naming.
5. Pick quant by hardware - default `Q4_K_M`, `Q5_K_M`/`Q6_K` for code/technical work, smaller `IQ`/`UD-*` variants for tight memory.
6. Convert with `convert_hf_to_gguf.py` + `llama-quantize` only when no GGUF exists; smoke-test the server's `/v1/chat/completions` endpoint.
Full skill & source: https://github.com/huggingface/skills/tree/ea9a24f38a98baf2430dab3bcb31a89753ae9e6d/skills/huggingface-local-models