Guide · Agent skill

How to use the Agent Eval skill to compare coding agents

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The agent-eval skill teaches your coding agent to set up and run agent-eval, a small command-line tool that runs several AI coding assistants on the same task and scores the results. It is aimed at developers and teams who want evidence, not impressions, before choosing a coding agent or model for their own codebase. After this guide you will know how to install the skill, describe a task, trigger a comparison with a natural request, and read the report.

Agent Eval (コーディングエージェント比較)affaan-m/ECC · MIT Download the skill

Quick start

1Install the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
2Review and install agent-eval. Read the source of the agent-eval repository at github.com/joaquinhuigomez/agent-eval, then install the CLI tool from there.
3Create a tasks folder. Run mkdir tasks and add one YAML task file per task, with a prompt, repo, files, judge checks and a pinned commit.
4Choose judges. Add at least one deterministic judge such as pytest or a build command, plus optional grep or llm judges.
5Run the agents. Run agent-eval run with --task, one or more --agent flags and --runs 3.
6Read the report. Run agent-eval report --format table and compare pass rate, cost, time and consistency.

What the agent-eval skill makes the agent do

Agent skills are folders of instructions that Claude reads when a request matches them. This one, described in its SKILL.md file, tells the agent how to use the agent-eval CLI to compare coding agents such as Claude Code, Aider and Codex head to head.

You define a task in a YAML file with a starting commit, a prompt and judge checks. The tool then runs each agent on that task and records whether it passed, what it cost, how long it took and how consistent it was across repeated runs.

Pass rate: whether the agent produced code that passes the judge.
Cost: API cost per task, where available.
Time: wall-clock seconds to finish.
Consistency: pass rate across repeated runs, for example 3/3 is 100%.

Install the agent-eval skill

Download the ZIP for this skill from this page and unzip it into one of these locations. For a personal skill, use ~/.claude/skills/. For a single project, use the .claude/skills/ folder inside that project.

You can also upload the skill in the Claude apps under Settings → Capabilities → Skills.

Note that the agent-eval tool itself is separate from the skill. The skill explains the workflow, but you must install agent-eval from its linked GitHub repository, and the source says to review that source first.

Personal: ~/.claude/skills/
Project: .claude/skills/ inside your project
Claude apps: Settings → Capabilities → Skills

Define a task in YAML

Each task is a YAML file that says what to do, which repository and files are involved, how success is judged, and which commit to start from. Pinning the commit makes results reproducible across days or weeks.

Keep your tasks in a tasks/ directory with one file per task. The fields shown in the source are name, description, repo, files, prompt, judge and commit.

Create the folder with mkdir tasks, then add one YAML file per task.
Use an existing commit hash in the commit field so every run starts from the same code.

name: add-retry-logic
description: Add exponential backoff retry to the HTTP client
repo: ./my-project
files:
  - src/http_client.py
prompt: |
  Add retry logic with exponential backoff to all HTTP requests.
  Max 3 retries. Initial delay 1s, max delay 30s.
judge:
  - type: pytest
    command: pytest tests/test_http_client.py -v
  - type: grep
    pattern: "exponential_backoff|retry"
    files: src/http_client.py
commit: "abc1234"

Run agents and compare the results

To run agents against a task, the skill uses this command, which runs each listed agent three times. For each run, the tool creates a new git worktree from the commit, passes the prompt to the agent, runs the judge checks, and records the result, cost and time. Each run gets its own git worktree, so no Docker is needed and runs do not affect each other or the base repository.

Then generate the comparison report. The source shows a table layout like the one below, with one row per agent.

agent-eval run --task tasks/add-retry-logic.yaml --agent claude-code --agent aider --runs 3
agent-eval report --format table

Choose judges that check results objectively

Judges decide whether a run passed. The source describes three types. Code-based judges are deterministic, such as running pytest or a build command. Pattern-based judges use grep to look for a pattern in files. Model-based judges use an LLM to assess the code against a written prompt.

The source recommends at least one deterministic judge per task, because LLM judges add noise. Use the llm type only when you need a judgement that tests cannot give.

Code-based: type pytest or type command, for example npm run build.
Pattern-based: type grep with a pattern and files.
Model-based: type llm with a prompt. This needs an API key.

judge:
  - type: command
    command: npm run build
  - type: grep
    pattern: "class.*Retry"
    files: src/**/*.py

Practical tips before you rely on results

The source advises starting with three to five tasks that represent your real workload, not toy examples. Run each agent at least three times, because agents are non-deterministic. Track cost alongside pass rate, since a very expensive agent may not be the right choice even if it passes more often. Keep task definitions under version control and treat them as test fixtures.

Start with 3 to 5 tasks drawn from real work.
Run each agent at least 3 times.
Pin the commit in every task file.
Include at least one deterministic judge per task.
Track cost with pass rate.
Version control your task definitions.

What people use it for

Choosing between Claude Code and Aider for a refactor in your own codebase.
Checking whether a newer model version regresses on a test-backed feature task.
Comparing the cost and pass rate of two coding agents before a team adopts one.
Building a small set of repeatable tasks to re-run whenever you change tools.

Frequently asked questions

Is the agent-eval skill free?

The source for this skill is published under the MIT licence. The agent-eval CLI is installed separately from its linked GitHub repository, and the coding agents and any API keys you use have their own costs and terms.

What do I need to install before using it?

You need the agent-eval CLI tool, git, the coding agents you want to compare with their own accounts or API keys, and pytest or npm if your judge commands use them. An API key is only needed for the LLM-as-judge option.

How do I install the agent-eval skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder. You can also upload it in the Claude apps under Settings → Capabilities → Skills.

Does agent-eval need Docker?

No. According to the source, each agent run gets its own git worktree, so Docker is not required for isolation.

Is agent-eval safe to run on my production repository?

The source warns against running it on sensitive or production repositories without review, because the agents and judge commands execute code and may incur API costs. Review the source first and try it on a non-critical repository.

How is this different from a standard benchmark?

It is not a turnkey benchmark. It measures agents on tasks you define for your own codebase, and the skill only explains the workflow, so you write the tasks yourself.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are a developer or maintainer who needs evidence about which coding agent or model works best on your own code, or who wants to catch regressions after upgrades. Skip it for quick one-off coding help, or if you want a ready-made public benchmark, since writing tasks and running repeated trials takes real effort.

Download the skill