Guide · Agent skill
How to use the Agentic Evaluation Patterns agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The agentic-eval skill is a set of reference patterns that teaches an AI agent to check and improve its own work through repeated self-review. Instead of returning a first draft, the agent can generate output, score it against criteria you define, note what fails, and revise, within a limited number of rounds. This guide explains what the skill changes, how to install it, how to trigger it, and where its limits are, so you can decide whether it fits your task.
Agentic Evaluation Patternsgithub/awesome-copilot · MIT Download the skill Quick start
- 1Download and unzip. Download the agentic-eval ZIP from this page and unzip it. Confirm that SKILL.md is in the extracted folder.
- 2Place the skill folder. Copy it into ~/.claude/skills/ for personal use, or into .claude/skills/ in a project. Alternatively, upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Define your criteria. Write down the measurable criteria and a score threshold that count as good enough for your output.
- 4Ask with a clear goal. Describe the task, the criteria and the maximum number of revision rounds in a normal request.
- 5Implement the helpers. Write the llm(), run_tests() or equivalent functions the examples call, since the skill does not supply them.
- 6Log and check results. Record each iteration and spot-check final outputs, especially where the stakes are high.
What is an agent skill, and what does agentic-eval change?
An agent skill is a folder containing a SKILL.md file with instructions that an AI agent can load when a task matches the skill's description. The skill tells the agent how to approach that kind of work.
The agentic-eval skill covers evaluation and self-improvement. Its SKILL.md describes a generate, evaluate, critique, refine cycle, and it includes Python-style examples of reflection loops, separate generator and evaluator roles, test-driven code fixing, and rubric scoring. It also includes a best-practices table and a quick-start checklist.
- Reflection loop: the agent critiques its own output against criteria and revises failing parts.
- Evaluator-optimizer: generation and evaluation are split into separate steps.
- Code-specific reflection: generated code is tested and fixed based on errors.
- Rubric and LLM-as-judge scoring: outputs are rated on weighted dimensions.
- Iteration limits: the patterns recommend a maximum of 3 to 5 rounds to avoid endless loops.
Install the agentic-eval skill from the ZIP
Download the agentic-eval ZIP from this page and unzip it. The skill ships as a single file, SKILL.md, so the extracted folder should contain that file.
For a personal install, place the skill folder in ~/.claude/skills/. To limit it to one project, place it in the project's .claude/skills/ folder instead.
You can also upload the skill in the Claude apps under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload
How to trigger the skill with natural requests
You usually do not need a special command. The skill's description lists the situations where it applies, so the agent loads it when your request matches. Describe the goal and the quality bar you care about.
Requests that match the skill's description include building self-critique loops, designing LLM-as-judge evaluation, adding iterative improvement to code or reports, and measuring agent response quality. Stating your criteria explicitly helps the agent apply the patterns well.
- "Add a self-review loop to our report generator that checks every figure against the source data."
- "Write a rubric for scoring our support replies on accuracy, clarity and completeness."
- "Set up a test-driven fix loop so the agent regenerates the parser until the pytest suite passes, with at most 3 attempts."
- "Build an evaluator-optimizer step that rejects summaries scoring below 0.8."
Example: a rubric-based evaluation
The skill's rubric pattern defines weighted dimensions and asks the model to rate each one from 1 to 5, then combines the scores. The weights below come from the skill's example.
The example calls an llm() helper and uses json.loads, so you need to supply those parts in your own code. The skill does not provide them.
RUBRIC = {"accuracy": {"weight": 0.4}, "clarity": {"weight": 0.3}, "completeness": {"weight": 0.3}}What you need before using it
The skill lists no required tools, packages or accounts. It does not run scripts and does not use the network by itself. Its code is illustrative: the examples call helpers such as llm() and run_tests() that are not defined in the file.
To use the patterns in a real pipeline, you or your agent must write those helpers. Plan for a model that can return structured JSON, and for a test runner if you use the code-fixing pattern.
- No listed requirements and no scripts included.
- Undefined helpers you must implement: llm(), run_tests(), and the parsing around them.
- Each loop round costs extra model calls and adds latency.
Safety notes and limits
LLM self-grading is not authoritative. The model's judgments can be inconsistent or wrong, so do not rely on it alone for high-stakes decisions. Keep a human or an independent check in those cases.
The skill's own checklist recommends adding convergence detection, logging every iteration, and handling evaluation parse failures gracefully.
- Do not treat self-grading as final for medical, legal, financial or safety decisions.
- Stop early if the score is not improving between rounds.
- Log each iteration so you can see why an output changed.
- Skip the loop for simple one-off tasks that have no clear success criteria.
What people use it for
- Adding a self-review pass to a code generator that runs tests and retries failed fixes.
- Scoring generated reports against a rubric for accuracy, clarity and completeness.
- Designing an LLM-as-judge step that compares two candidate outputs against a stated criterion.
- Building an evaluator-optimizer pipeline that revises summaries until they meet a score threshold.
Frequently asked questions
How do I install the agentic-eval skill?
Download the ZIP from this page and unzip it. Put the folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder, or upload it in the Claude apps under Settings → Capabilities → Skills.
Does agentic-eval need any requirements or accounts?
The skill lists no required tools or packages. It does not run scripts and does not use the network. The code examples, however, depend on helper functions like llm() and run_tests() that you must write yourself.
Is agentic-eval free to use?
The skill is published under the MIT licence, as listed on this page. Check the licence file in the source repository for full terms.
Will this skill run an evaluation framework for me?
No. The file is guidance and code sketches, not a runnable evaluation framework. You will need to implement the helpers and wire the loop into your own pipeline.
How is this different from just asking the model to double-check its answer?
The skill sets out structured patterns: explicit criteria, scoring, a limited number of revision rounds, convergence checks and logging. A plain double-check has none of those controls, which makes the results harder to measure or debug.
Why does my agent keep looping or not improving?
The skill recommends an iteration limit of 3 to 5 and a convergence check that stops when the score is no longer improving. Add both, and make sure your criteria are specific enough to measure.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use agentic-eval if you are building agent pipelines that need repeatable quality checks on code, reports or analysis, and you can supply the missing helper code. Pick something else if you want a ready-made evaluation tool, need a simple one-off task, or would rely on LLM self-grading as the final word on high-stakes decisions.
Download the skill