Agentic Evaluation Patterns
Teaches an agent to critique and iteratively refine its own outputs using rubrics and test loops.
- Repository
- github/awesome-copilot
- Stars
- 40,703
- Licence
- MIT
- Files
- 1 file
What it does
When installed, the agent gets a set of reference patterns for making its work better through repeated self-review. Instead of producing one answer and stopping, it can generate a draft, score it against criteria you define, note what fails, and revise, repeating a limited number of times. The file includes Python-style examples of reflection loops, separate generator and evaluator roles, test-driven code fixing, and rubric scoring, plus a best-practices table and a checklist.
Great for
- Developers building agent pipelines that need quality checks on generated code or reports
- Teams designing LLM-as-judge or rubric-based evaluation for agent outputs
- Anyone who wants an agent to self-check against clear, measurable criteria before delivering results
Skip it if
- Simple one-off tasks with no clear success criteria, where the extra loops add cost and latency without benefit
- Users who expect a ready-made tool: the file is guidance and code sketches, not a runnable evaluation framework, and the examples call undefined helpers like llm() and run_tests()
- Situations where LLM self-grading is treated as authoritative for high-stakes decisions, since the model's judgments can be inconsistent or wrong
Try asking
- “Add a self-critique loop to my report generator that checks each section against our style guide”
- “Design a rubric with weighted dimensions for scoring our support chatbot's answers”
- “Build a test-driven refinement workflow where generated Python code is fixed until pytest passes”
- “Set up an evaluator-optimizer pipeline with an iteration limit and convergence check”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
agentic-eval.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip agentic-eval.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip agentic-eval.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Define concrete, measurable criteria before asking the agent to evaluate, since vague criteria produce vague critiques
- Keep iteration limits low (about 3 to 5) and add a convergence check so loops stop when scores stop improving
- Treat the example code as a starting sketch: you must supply the model call, JSON parsing, and test runner, and handle parse failures
What's inside
- SKILL.md
Redistributed unmodified from github/awesome-copilot under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.