How to use the Agent Evals agent skill (agent-evals)

What does the Agent Evals skill make the agent do?

The skill is about building test infrastructure, not answering questions. When it is active, the agent writes evaluation suites that run repeatedly so you can check whether an agent or prompt change made behaviour better or worse.

The source describes several layers: unit evals for individual prompt-to-response quality, tool evals for whether the agent picks the right tool with the right parameters, end-to-end evals for multi-step tasks, safety evals for adversarial prompts, and LLM-as-judge scoring for free-form outputs. The templates are meant to run as regression gates before you deploy prompt or agent changes.

• Unit evals: check responses against must-contain and must-not-contain keywords or exact JSON output
• Tool evals: check that expected tools are called and expected parameters match
• End-to-end evals: check multi-step runs, such as whether search was used and a token budget was respected
• Safety evals: check that injection prompts do not leak a system prompt or secrets
• LLM-as-judge: a grading model scores accuracy, helpfulness and safety on a 1 to 5 scale

How do you install the Agent Evals skill?

Download the ZIP file from this page. Unzip it into one of these locations, depending on how you want to use it: your user skills folder at ~/.claude/skills/, or a project's .claude/skills/ folder if you want the skill available only inside that project.

If you use the Claude apps instead, upload the skill under Settings, then Capabilities, then Skills.

The package contains a single file, SKILL.md. The source does not include a separate install command, so do not assume one exists.

• Option A: unzip into ~/.claude/skills/ for personal use
• Option B: unzip into a project's .claude/skills/ folder to scope it to one repository
• Option C: upload it in the Claude apps under Settings, then Capabilities, then Skills

How do you trigger the skill with natural requests?

You do not need a special command. The skill's description says to use it when shipping agent features, validating prompt changes, or gating deployments on quality. Ask for those tasks in plain language and the agent should pick it up.

Be specific about the thing you want tested. Name the agent or feature, the kind of checks you need, and where the output should go. The source's own use cases include building regression suites for tool-calling agents, adding CI gates for agent quality and safety, measuring LLM output quality at scale, and validating RAG retrieval accuracy.

• Set up regression tests for our support agent's tool calls before we change the prompt
• Add a CI gate that blocks deployment if the agent's safety score drops
• Build a golden dataset of 20 expected answers for our RAG retrieval and a pytest file to run it
• Write prompt-injection tests for our chatbot that check it does not reveal its system prompt

What examples and dataset format does the skill use?

The core idea is a golden dataset: a JSON list of cases, each with a prompt and the properties a good answer must have. The source gives this example, shown here with formatting removed. The first case checks a tip calculation for a must-contain value, and the second checks that a prompt-injection attempt is refused.

The templates load cases like these from files such as evals/fixtures/unit_cases.json. Keep fixtures synthetic, as the source advises.

• Each case has an id, a prompt, and optional must_contain, must_not_contain or expected_json fields
• Tool cases can specify expected_tool and expected_params_subset
• Judge cases include a reference answer that the grading model compares against

What do you need before running these evals, and what are the safety limits?

You need Python 3.10 or later, an LLM API key from a provider such as OpenAI or Anthropic, pytest, and pytest-timeout, which the shown --timeout options require. You also need an agent adapter, fixture files and reviewed dependencies, which you supply yourself. The templates are not bundled executable tests. Optionally, you can connect a Braintrust, Promptfoo or LangSmith account.

The skill's metadata says it does not run scripts and does use the network. Calls to the model provider send your prompts and data off your machine and cost money, so approve that before running. Keep API keys private. The source also says to sandbox tool effects and to inspect tool and network traces, not just the response text.

The frontmatter compatibility note also lists platform CLIs such as kubectl, helm, terraform, git and CI runners. Those are not part of the Python requirements above, so check whether they apply to your setup.

• Approve provider cost and data transmission before running
• Keep fixtures synthetic and API keys private
• Sandbox tool effects and inspect tool and network traces
• Define thresholds and human review before you evaluate
• Keyword refusals and numeric judge scores do not prove safety or factual accuracy

What tips help when adapting the templates?

Treat the templates as starting points. The source says you must supply the agent adapter, the fixtures, supported provider and model IDs, and the test configuration. The example model IDs in the templates are illustrative, so use IDs your provider actually supports.

Decide thresholds before you run the suite. The judge template, for example, fails if safety falls below 4 or average accuracy falls below 3.5, but whether those numbers suit your product is your call. Pair automated scores with human review.

• Replace the placeholder agent import with your own adapter
• Write your own golden cases from real user queries
• Set thresholds and review process before you treat a pass as meaningful
• Use the optional Promptfoo section of the source for a config-based setup, if you use Promptfoo