Runs reproducible model evaluations against Hugging Face Hub models on local hardware using inspect-ai or lighteval, with guidance on choosing vLLM, Transformers, or accelerate backends.
---
name: huggingface-community-evals
description: Run evaluations for Hugging Face Hub models on local hardware with inspect-ai and lighteval. Use for backend selection (vLLM / Transformers / accelerate), local GPU evals, and smoke-testing benchmarks before scaling. Not for HF Jobs orchestration or results publishing.
---
# Hugging Face Community Evaluations
Run benchmark evaluations against Hub models locally, choosing the right framework and inference backend.
## Workflow
1. Pick the framework: `inspect-ai` for explicit task control, `lighteval` for leaderboard-style task strings (`suite|task|num_fewshot`).
2. Pick the backend: prefer `vllm` for throughput; fall back to Transformers (`--backend hf`) or `accelerate` for unsupported architectures; use the providers-backed `inspect_eval_uv.py` when you don't need direct GPU control.
3. Verify prerequisites with `uv --version`, `printenv HF_TOKEN`, and `nvidia-smi`.
4. Always start with a smoke test (`--limit 10` for inspect-ai, `--max-samples 10` for lighteval), then scale up.
5. Run scripts via `uv run scripts/inspect_eval_uv.py` / `inspect_vllm_uv.py` / `lighteval_vllm_uv.py`.
6. For remote compute, hand off the same script and args to the `hugging-face-jobs` skill.
Full skill & source: https://github.com/huggingface/skills/tree/ea9a24f38a98baf2430dab3bcb31a89753ae9e6d/skills/huggingface-community-evals