Skip to content
Get packs

Guide · Agent skill

How to use the LLM Benchmark Harness agent skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The evaluating-llms-harness skill teaches an AI agent to set up and run EleutherAI's lm-evaluation-harness, a widely used tool that tests language models on 60+ academic benchmarks such as MMLU, GSM8K, HellaSwag and HumanEval. This guide is for people who may be new to agent skills and want to install the skill, ask for an evaluation in plain language, and understand what the agent will run. By the end you will know how to install the skill, what it needs, and how to get reproducible benchmark results saved to files.

LLM Benchmark HarnessOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Install the skill. Download the ZIP from our page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
  2. 2Install lm-eval. Run pip install lm-eval in a Python environment. Confirm you have a CUDA GPU if you plan to evaluate HuggingFace models locally.
  3. 3Get model access. For gated models such as Llama-2, create a HuggingFace account and request access before the run.
  4. 4Ask for an evaluation. Describe the model and the benchmarks in plain language, for example MMLU and GSM8K for a release report.
  5. 5Review the command. Check the lm_eval command the agent proposes, including model, tasks, device and output path, before running it.
  6. 6Run and save results. Let the run finish, then check the JSON file in your output folder for each task's scores and the config used.

What does the skill make the agent do?

Without the skill, an agent has no reliable knowledge of how to run lm-evaluation-harness. With it, the agent can write the correct lm_eval command-line invocations, choose suitable tasks and settings, save results to files, and help compare models or track scores across training checkpoints.

The skill runs an existing tool. It is a guide for benchmarking, not a new evaluation method.

  • Installs the lm-eval package with pip install lm-eval
  • Runs HuggingFace models, with vLLM or API backends also supported
  • Runs standard suites such as mmlu, gsm8k, hellaswag, truthfulqa and arc_challenge
  • Writes results to JSON files, and can save individual predictions with --log_samples

How do you install the evaluating-llms-harness skill?

Download the ZIP from our page and unzip it. For a personal setup, place the folder in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder.

In the Claude apps you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal: ~/.claude/skills/
  • Project: .claude/skills/ inside your project
  • Claude apps: Settings → Capabilities → Skills

How do you trigger the skill with a natural request?

You do not need special syntax. Describe the job in plain language and mention benchmarking, comparing models, academic results or training progress. The skill's description matches requests like these.

Name the model and the benchmarks if you know them. Otherwise the agent will suggest a standard suite.

  • "Benchmark my fine-tuned Llama checkpoint on MMLU and GSM8K and save the results."
  • "Compare these four open models on the standard academic suite and give me a table."
  • "Set up a 0-shot GSM8K and HellaSwag check I can run every 1000 steps during training."

What do you need before running an evaluation?

You need Python with the lm-eval package. A GPU (CUDA) is recommended for HuggingFace models. Gated models such as Llama-2 require a HuggingFace account and access approval before they can be downloaded. vLLM or API access is optional and only needed for those backends.

Plan for run time. The source notes that HellaSwag takes about 10 minutes on one GPU, GSM8K about 5 minutes, and MMLU about 2 hours across 57 subjects.

  • Python with pip install lm-eval
  • A CUDA GPU recommended for HuggingFace models
  • HuggingFace access, including approval for gated models
  • Optional: vLLM or API access for hosted models
pip install lm-eval

What does a typical evaluation look like?

A standard run passes the model, the tasks, the device and the batch size. Adding --num_fewshot 5 uses the 5-shot setting the source describes as standard, and --output_path writes results to a file. To see every available task, ask for the task list.

The results file records each task's scores and the configuration used, so you can rerun or compare later.

  • Use --output_path results/ to save output and --log_samples to save individual predictions
  • Use --batch_size auto to let the tool pick a batch size on HuggingFace models
  • Use --num_fewshot 0 for faster checks during training
lm_eval --model hf \ --model_args pretrained=meta-llama/Llama-2-7b-hf \ --tasks mmlu,gsm8k,hellaswag \ --device cuda:0 \ --batch_size 8 lm_eval --tasks list

How do you compare models or track training progress?

For comparisons, the skill's workflow runs the same task list against each model and writes one results file per model, then builds a table from the primary metric of each task. For training, it runs a quick benchmark set on each saved checkpoint and plots the scores against the training step.

Use the same tasks, few-shot count and batch settings for every model in a comparison, otherwise the numbers are not directly comparable.

  • Compare: one results file per model, then one table
  • Track training: quick tasks such as gsm8k and hellaswag per checkpoint, then plot the curve
  • Avoid MMLU and HumanEval for frequent checks because they are slow

What safety points should you know about?

This skill runs scripts and uses the network. Downloading gated models and datasets goes over the network, and the commands can load model weights from HuggingFace or a local path. Review commands before they run, especially on shared machines.

HumanEval-style tasks execute generated code. Do not run them without a sandbox, and do not run untrusted code-generation tasks on a machine with sensitive data.

  • Runs scripts and uses network access
  • Review generated shell commands before running them
  • Run code-execution tasks such as HumanEval inside a sandbox
  • Public benchmarks may be contaminated in training data, so they do not replace tests on your own tasks

What people use it for

  • Reporting standard benchmark scores for a model release
  • Comparing several open models on the same academic tests
  • Tracking a model's quality across training checkpoints with quick tasks
  • Checking a quantized model against its full-precision version on MMLU or GSM8K
  • Building a custom domain task from YAML, using the skill's custom-tasks reference

Frequently asked questions

How do I install the evaluating-llms-harness skill?

Download the ZIP from our page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps you can upload it under Settings → Capabilities → Skills.

What do I need to run lm-evaluation-harness?

You need Python with the lm-eval package, installed with pip install lm-eval. A CUDA GPU is recommended for HuggingFace models, and gated models such as Llama-2 need a HuggingFace account and access approval.

Is lm-evaluation-harness free to use?

The skill is listed under the MIT licence. The lm-evaluation-harness tool is an open tool installed via pip; check its own repository for its licence terms before commercial use.

Which benchmarks does the skill cover?

The source describes 60+ academic benchmarks, including MMLU, GSM8K, HellaSwag, TruthfulQA, ARC and HumanEval. Run lm_eval --tasks list to see the full set available to your install.

Can it evaluate API models like GPT or Claude?

Yes. The skill's API reference covers OpenAI and Anthropic models, plus local OpenAI-compatible endpoints, through a unified TemplateAPI interface.

Why is my evaluation so slow?

MMLU across 57 subjects can take around 2 hours, and larger models need more memory and time. Use quick tasks such as gsm8k and hellaswag for frequent checks, and --batch_size auto on HuggingFace models.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are an ML engineer or researcher who needs standard, reproducible academic benchmark scores for a model release, a comparison, or a training run. Skip it if you mainly want to judge a model on your own real-world tasks, since public benchmarks may not reflect them, or if you have no GPU or cloud budget for long runs.

Download the skill