How to use the Eval Harness First agent skill

What is an agent skill and what does this one make the agent do?

An agent skill is a folder containing a SKILL.md file and optional reference files. When it is installed, the agent reads the skill's instructions when a request matches the skill's description, and follows its process. The skill's own name is eval-harness-first and its description says it is used when starting a fine-tuning effort, converting traces into an eval set, or calibrating a judge against human labels.

The skill's core rule is that the harness must exist before any training config is written. The agent works through a flywheel: collect traces (or create synthetic tasks if none exist), run error analysis by open-coding at least 100 traces and grouping them into 4 to 8 failure buckets, write one grader per bucket, prioritize by frequency, severity and value, and then record a baseline for the base model. The same labeled traces feed training data, minus an explicit holdout of evaluation IDs.

• Collects or creates example tasks, using production traces when they exist and a task spec when they do not.
• Groups failures into named categories through open coding and axial coding.
• Writes one grader per category, preferring regex, schema or execution checks over AI judges.
• Calibrates any AI judge against human labels, reporting true positive and true negative rates.
• Records a baseline score for the unmodified base model in eval/baseline-<model>.json.

Install the eval-harness-first skill

Download the skill's ZIP from the item page on this site. Then choose one of two install locations. For a personal install available in every project, unzip the folder into ~/.claude/skills/. For a single project, unzip it into that project's .claude/skills/ folder. The skill folder should contain SKILL.md and the references/ folder.

If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills. Use the same ZIP file.

• Personal: unzip into ~/.claude/skills/
• Project only: unzip into your project's .claude/skills/
• Claude apps: Settings → Capabilities → Skills, then upload the ZIP

Prepare what the skill needs before you start

The skill assumes you have production or agent traces. If you do not, you need a task spec describing the work. You also need human labelers who can grade at least 100 examples. The skill's reference files for grader templates and judge calibration are included in the ZIP, so you do not need to fetch them separately.

If you cannot meet these requirements, the skill's own guidance is that the baseline would be unreliable. In that case, a simple prompt change or a project that will not fine-tune a model is a better fit for a different approach.

• Production or agent traces, or a task spec if none exist
• Human labelers able to grade at least 100 examples
• The included references/grader-templates.md and references/judge-calibration.md files

How to trigger the skill with natural requests

You do not need a special command. Describe what you are doing in plain language, and mention fine-tuning, evaluation, traces or judges so the agent matches the skill's description. The skill is designed for three situations: starting a fine-tuning effort, converting traces into an eval set, and calibrating a judge against human labels.

The skill's text also names the output it produces: an eval/ directory with goldens, graders, a drift suite and a base-model baseline. Asking for those outputs by name is a clear way to trigger it.

• "Before we fine-tune, help me build an evaluation harness for our support agent."
• "Turn these production agent traces into a labeled eval set."
• "Help me calibrate our LLM judge against human labels before we trust its verdicts."
• "Record a baseline for the unmodified base model before we start training."

Example workflow

Suppose you run an agent that handles customer requests and you want a fine-tuned model. Ask the agent to start the harness from your exported traces. The skill directs you to open-code at least 100 traces, group failures into buckets, and write a grader for each bucket. Where a failure is about output shape, the skill's schema-compliance approach applies. Where a criterion such as tone is genuinely subjective, the skill routes it to a judge, which must then be calibrated.

Once the buckets, graders and goldens are in place, the skill asks for the baseline run against the base model. The resulting baseline file is what later checkpoints are compared against.

Safety notes: scripts and network

The skill does not run scripts and does not use the network, according to its item details. Its grader templates are Python examples that you wire into your own pipeline, so review any code before you run it against your data. Keep evaluation data out of the training set by ID, as the skill instructs, and never let run scripts write into the eval/ directory.

• Runs scripts: no
• Uses network: no
• Review the Python grader examples before running them on your data

Tips for getting useful results

Use one grader per failure bucket instead of one blended score, because a single score hides which bucket regressed. Prefer binary pass/fail over 1-to-5 scales. Commit eval/goldens.jsonl to version control and tag it per release so it can serve as a regression suite. Pin any judge to a fixed model snapshot and recalibrate when the judge model changes or every quarter.

The judge must come from a different model family than the model under test. A judge that misses the agreed true positive and true negative bar should only flag items for human review, not gate promotions.

• Start with deterministic checks and add a judge only for subjective criteria.
• Split labeled judge data into train, dev and sealed test sets.
• Report true positive and true negative rates, not one accuracy number.