How to use the Hugging Face Local Model Evals agent skill

What an agent skill is and what this one does

An agent skill is a folder containing a SKILL.md instruction file and supporting files, which an AI agent loads when a request matches the skill's description. This skill's SKILL.md tells the agent how to run evaluations on Hub models using inspect-ai and lighteval on local hardware.

The skill covers choosing between the vllm, Hugging Face Transformers and accelerate backends, smoke tests, task selection and backend fallback. It does not cover Hugging Face Jobs orchestration, model-card or model-index edits, README table extraction, .eval_results generation or publishing, PR creation, or community-evals automation.

• Local inspect-ai eval on a Hub model via inference providers (scripts/inspect_eval_uv.py)
• Local GPU eval with inspect-ai using vllm or Transformers (scripts/inspect_vllm_uv.py)
• Local GPU eval with lighteval using vllm or accelerate (scripts/lighteval_vllm_uv.py)
• Extra command patterns in examples/USAGE_EXAMPLES.md

Install the huggingface-community-evals skill

Download the skill as a ZIP from the skill's page on this directory and unzip it. Put the resulting folder in one of two places: ~/.claude/skills/ to make it available across your projects, or a project's .claude/skills/ folder to limit it to that project.

If you use the Claude apps instead, upload the ZIP under Settings → Capabilities → Skills.

• Personal install: ~/.claude/skills/
• Project install: .claude/skills/ inside your project
• Claude apps: Settings → Capabilities → Skills

Check the requirements before your first run

The skill expects uv (the Python package runner) and, for local GPU runs, an NVIDIA GPU with nvidia-smi available. Set the HF_TOKEN environment variable if you need gated or private models. The scripts declare their Python packages in their headers: inspect-ai, inspect-evals, openai, vllm, torch, transformers, lighteval and accelerate.

Run these checks before starting a local GPU evaluation:

Trigger the skill with a natural request

You do not need to name a script. Ask the agent in plain language, for example to run MMLU on a Hub model on your GPU, or to compare vLLM and Transformers for a model. The skill's description covers backend selection, local GPU evals and choosing between vLLM, Transformers and accelerate, so requests on those topics should match it.

Be specific about the model ID, the benchmark and whether you want a quick test. Requests about publishing scores or running on remote cloud jobs fall outside this skill and will be handed off or declined.

• Run a quick MMLU smoke test on meta-llama/Llama-3.2-1B using my local GPU.
• Evaluate microsoft/phi-2 on GSM8K, and fall back to Transformers if vLLM doesn't support it.
• Which backend should I use for a 3B model on my GPU: vLLM, Transformers or accelerate?

Run a smoke test first, then scale up

The skill's core workflow is to pick a framework, pick a backend, run a small smoke test, and only then scale up. For inspect-ai, use --limit 10 or similar. For lighteval, use --max-samples 10. A small run confirms the model loads and the task runs before you spend GPU time on a full evaluation.

The first command below uses the inspect-ai path with inference providers, which needs no direct GPU control. The second runs inspect-ai on a local GPU with vLLM.

Run lighteval and choose a backend

Use lighteval when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks. Task strings use the format suite|task|num_fewshot, and several can be comma-separated in --tasks. Prefer vLLM for throughput on supported architectures, and use accelerate as the compatibility fallback.

For inspect-ai, use --backend hf with Transformers when vLLM does not support the model. The Transformers fallback in the example below also uses --trust-remote-code, which runs custom code from the model repository (see the safety section).

• Task string examples: leaderboard|mmlu|5, leaderboard|gsm8k|5, leaderboard|arc_challenge|25, lighteval|hellaswag|0
• inspect-ai task examples: mmlu, gsm8k, hellaswag, arc_challenge, truthfulqa, winogrande, humaneval

Pick hardware and fix common errors

The skill's guidance is that models under 3B suit a consumer GPU, Apple Silicon or a small development GPU; 3B to 13B need a stronger local GPU; and 13B and above need a high-memory local GPU or a handoff to hugging-face-jobs. For out-of-memory errors on CUDA or vLLM, reduce --batch-size or --gpu-memory-utilization, or try a smaller model for the smoke test.

If vLLM does not support a model, switch to --backend hf for inspect-ai or --backend accelerate for lighteval. If a gated or private repo fails to load, check HF_TOKEN. If the model needs custom code, add --trust-remote-code.

• CUDA or vLLM out of memory: reduce --batch-size, reduce --gpu-memory-utilization, or use a smaller smoke-test model
• Model unsupported by vLLM: use --backend hf (inspect-ai) or --backend accelerate (lighteval)
• Gated or private repo fails: verify HF_TOKEN
• Custom model code required: add --trust-remote-code

Safety notes: scripts, network and remote code

This skill runs scripts and uses the network, so the agent will execute the three provided Python scripts through uv and download models and datasets. Review the scripts and the arguments before running them, especially on a machine with sensitive files.

The Transformers fallback can use --trust-remote-code, which executes custom code from the model repository. Only use that flag with models whose code you trust.

• The skill runs scripts and uses network access.
• --trust-remote-code executes code from the model repository; avoid it for untrusted models.
• Keep HF_TOKEN in your environment rather than pasting it into prompts.