Eval Harness First
Build the measurement setup (test sets, graders, judge checks, baselines) before any fine-tuning run.
- Repository
- wshobson/agents
- Stars
- 42,095
- Licence
- MIT
- Files
- 3 files
What it does
When installed, the agent guides you through building an evaluation harness before fine-tuning a model. It helps collect or create example tasks, group failures into categories, write a grader for each category (preferring simple automated checks over AI judges), calibrate any AI judge against human labels, and record a baseline score for the unmodified base model. The skill's main point is that this harness should exist first, because it both measures progress and supplies labeled data for training.
Great for
- Teams starting a fine-tuning project who want a measurable pass/fail gate before training
- ML engineers converting production agent traces into a labeled evaluation set
- Developers who need to calibrate an LLM judge against human labels
Skip it if
- Simple prompt tweaks or projects that will not fine-tune a model, since the process assumes a full training pipeline and labeling effort (at least 100 labeled examples)
- Teams without labelers or enough traces, because the skill's requirements cannot be met and the baseline would be unreliable
Try asking
- “Help me set up an eval harness before we fine-tune our support-ticket classifier.”
- “Turn these 150 production agent traces into a labeled golden set with failure buckets.”
- “Calibrate our LLM judge against 100 human labels and report TPR and TNR.”
- “Write the baseline eval for the base model before Phase 1 starts.”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 3 original files, the MIT licence and an attribution note.
eval-harness-first.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip eval-harness-first.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip eval-harness-first.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Start by open-coding at least 100 real traces before choosing any grader, so the failure buckets come from evidence.
- Keep eval/goldens.jsonl excluded from training data by ID, or the harness will be contaminated.
- Note that the skill references a /finetune command and checkpoint-promotion skill; those must exist in your setup for the full gate to work.
What's inside
- references/grader-templates.md
- references/judge-calibration.md
- SKILL.md
Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.