Skip to content
Get packs

Agent Evals

Build automated test suites that check whether an AI agent's answers, tool use and safety hold up over time.

Source Download skill
Repository
sickn33/agentic-awesome-skills
Stars
49,023
Licence
MIT
Files
1 file

What it does

This skill teaches an AI agent how to set up repeatable quality checks for other AI agents or LLM features. It provides Python test templates for prompt-level correctness, tool selection, multi-step tasks, prompt-injection resistance, and LLM-as-judge grading, plus a golden dataset format of expected answers. The templates are meant to run as regression gates before deploying prompt or agent changes.

Great for

  • Engineers shipping an LLM feature who want automated checks before each release
  • Teams adding CI gates that block deployments when agent quality or safety drops
  • Developers building regression suites for tool-calling agents or RAG retrieval

Skip it if

  • People who only want a one-off answer to a question; the skill is about building test infrastructure, not answering
  • Teams expecting turnkey tests: the file states the templates are not bundled executable tests and need an adapter, fixtures and config supplied by you
  • Anyone treating keyword checks or numeric judge scores as proof of safety or factual accuracy, which the skill itself warns against

Try asking

  • “Create a golden dataset and pytest suite to check my support bot's answers before each deploy”
  • “Write tool-selection evals that confirm my agent calls the right search function with the right parameters”
  • “Set up an LLM-as-judge rubric to score summaries for accuracy and helpfulness”
  • “Add prompt-injection tests to my CI pipeline to check the agent does not leak its system prompt”
Step-by-step guide: how to use Agent EvalsInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 1 original files, the MIT licence and an attribution note.

    agent-evals.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip agent-evals.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip agent-evals.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Review every generated test and fixture before running; the templates call real model providers and cost money.
  • Use synthetic fixtures and keep API keys out of the repository, since the tests send prompts and data to providers.
  • Define pass thresholds and human review rules before running judge-based scores, and sandbox tools so tests cannot cause real side effects.

What's inside

  • SKILL.md

Redistributed unmodified from sickn33/agentic-awesome-skills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.