Skip to content
Get packs

Advanced LLM Evaluation

Teaches the agent to judge LLM outputs reliably using rubrics, pairwise comparisons and bias controls

Source Download skill
Repository
muratcankoylan/Agent-Skills-for-Context-Engineering
Stars
18,282
Licence
MIT
Files
6 files

What it does

This skill teaches an AI agent how to grade the quality of AI-generated answers, using another AI model as the judge. It covers two main methods: scoring a single response against a rubric, and comparing two responses head to head. It also gives the agent known countermeasures for judging biases such as favoring longer answers or whichever answer appears first, and guidance on picking metrics and checking agreement with human reviewers.

Great for

  • Engineers building automated quality checks for chatbots or LLM features
  • Teams running A/B tests of prompts or models who need consistent grading
  • Researchers calibrating rubrics and comparing automated judgments with human ratings

Skip it if

  • Simple pass/fail tests or regression suites for deterministic code, which the related 'evaluation' skill covers better
  • Anyone expecting LLM judgments to be ground truth: the skill itself notes judges can carry systematic biases and should be validated against humans
  • Autonomous loops that need locked rubrics or approval gates, which the 'harness-engineering' skill handles

Try asking

  • “Compare these two chatbot answers and tell me which is better, controlling for position bias”
  • “Write a 1-5 scoring rubric for judging the helpfulness of customer support replies”
  • “Our LLM judge disagrees with human raters on tone; help me find the systematic pattern and calibrate it”
  • “Set up a pairwise evaluation pipeline with swapped-position consistency checks”
Step-by-step guide: how to use Advanced LLM EvaluationInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 6 original files, the MIT licence and an attribution note.

    advanced-evaluation.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip advanced-evaluation.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip advanced-evaluation.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent your actual rubric criteria and sample outputs so it can tailor the prompts instead of using the generic templates
  • Validate any judge against a small set of human-labelled examples before trusting its scores
  • Use a different model for judging than for generating, to reduce self-preference bias

What's inside

  • references/bias-mitigation.md
  • references/evaluation-pipeline.md
  • references/implementation-patterns.md
  • references/metrics-guide.md
  • scripts/evaluation_example.py
  • SKILL.md

Redistributed unmodified from muratcankoylan/Agent-Skills-for-Context-Engineering under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.