How to use the Calibrate skill to tune an AI judge to your taste

What does the Calibrate skill make the agent do?

Calibrate takes a task described in plain words, finds the eval suites and configured targets that cover it, and offers them as options. It then shows you N real outputs, images or text, with the default being 16. On the review page the outputs are shown blind, with no judge scores visible. You score each one from 0 to 10 and can add typed notes.

From those labels the agent measures how well the judge agrees with you, distills a short taste profile from your notes, and rescores the outputs to show whether the change helped. The judge reads the taste profile on later runs.

• Collects a 0 to 10 score per output plus optional notes
• Measures judge-versus-you agreement
• Writes a taste profile the judge reads on later runs
• Rescores to check whether the change improved agreement

Install the Calibrate skill

Download the ZIP for the Calibrate skill from this page. To install it for your user account, unzip it into ~/.claude/skills/. To install it for one project only, unzip it into that project's .claude/skills/ folder.

In the Claude apps you can instead upload the ZIP under Settings, then Capabilities, then Skills.

• Personal install: ~/.claude/skills/
• Project install: the project's .claude/skills/ folder
• Claude apps: Settings, Capabilities, Skills, then upload the ZIP

What you need before it works

Calibrate is not useful on its own. It depends on the LifeOS framework layout, with eval suites and a stored results folder at LIFEOS/MEMORY/STATE/Evals-Results/. The scripts are run with the Bun runtime. For image intents you also need a configured calibration target, which the AddTarget intent creates.

Optional Config.md and PREFERENCES.md files hold your own settings. The installer places starters for both. Setup.md in the skill explains the keys.

• Bun runtime, since scripts are run with bun
• LifeOS layout with eval suites and LIFEOS/MEMORY/STATE/Evals-Results/
• For image intents, a target at Targets/<target>/target.json created by AddTarget
• Optional Config.md and PREFERENCES.md, described in Setup.md

How to trigger the skill with natural requests

You do not need a command name. Ask in plain words. The skill's trigger phrases include calibrate, calibrate 16, calibrate 64, calibrate social posts, calibrate the art judge, calibrate this eval, show me outputs to rate, rate these images, teach the eval what I like, tune the eval to my taste, and calibration session.

Other requests route to different intents. Asking how well the judge matches you gets a report. Asking to update the taste profile runs Distill. Asking to add a calibration target for a new image intent runs AddTarget.

• calibrate 24 social posts: a session with 24 outputs
• show me outputs to rate: starts a session on the default count
• what does the judge get wrong about my taste: runs a report
• update the taste profile: runs Distill
• calibrate a new image intent: runs AddTarget

Run a session and read the results

A session serves the review page on your own machine and waits until you submit. Your scores and notes become labels. The report then compares the judge's verdicts with your labels. A score of 7 or higher counts as a like for pass/fail agreement, unless the target sets a different likeAt value.

The tool also reports the judge's cut-off that would best match your likes, and each disagreement next to your note. Treat held-out numbers as the result. In-sample numbers are only context. Calibrate never changes a bar or promotes a judge on its own. That needs your yes.

• Score each output 0 to 10 and add notes where they matter
• The report shows agreement and Cohen's kappa against shared floors
• Disagreements appear beside your own notes
• Nothing changes until you approve it

Useful commands and their purpose

The skill wraps a small command-line tool. You can run the commands yourself to check state, though the agent usually runs them for you. Run the commands from your config root.

• discover: ranks targets and suites against a task in plain words
• targets: lists configured targets
• pool: shows pool size, scored share, labeled share and score bands
• session: serves the review page and blocks until you submit
• report: shows judge-versus-you agreement
• taste-input: lists current labels with notes for Distill
• taste: validates citations and writes Taste.json
• rescore: reruns the target's judge with the current taste

Safety notes: scripts and network

Calibrate runs scripts, so read what you install and run it in an environment you trust. The skill's own metadata says it does not use the network. The review page is served on loopback, meaning your own machine, and the session opens it with the browser named in Config.md, or your system default browser if you have no Config.md.

Your labels, sessions and taste profile are stored under your Claude config directory, in the Calibrate folder.

• Runs scripts: yes, executed with bun
• Uses network: no, according to the skill metadata
• Review page is served on loopback
• Labels and taste profiles are stored locally under the Calibrate folder

Tips for better calibration

Use a pool of real published outputs rather than candidates, and aim for a large pool across several sessions. Give honest scores on every output, and add a note when a score matters. Notes are what the taste profile is built from, so specific notes produce a more useful profile.

Re-run the report after each taste update to see whether agreement improved on held-out labels.

• Score honestly; the judge only learns from what you actually give
• Add notes on surprising scores
• Keep a large pool of real outputs
• Check held-out agreement, not only in-sample numbers