Skip to content
Get packs

Calibrate Judge to Taste

Shows real outputs, collects your 0–10 scores, and tunes an eval judge to match your taste.

Source Download skill
Repository
danielmiessler/LifeOS
Stars
19,682
Licence
MIT
Files
9 files

What it does

Calibrate lets an AI agent check whether its automated quality judge (the part that grades outputs) agrees with you. It shows you a set of real outputs on a local review page, you score each one from 0 to 10 and can add notes, and the agent then measures how often the judge agrees with you. It writes a short taste profile from your notes that the judge reads on later runs, then re-scores to show whether the change helped.

Great for

  • Teams running an eval or LLM judge who want it to reflect one person's or one brand's taste
  • Creators or designers tuning how an AI art or writing judge grades their work
  • Developers building or maintaining Assay-style eval suites who need human-labelled calibration data

Skip it if

  • Simply producing images or text: this skill grades and tunes judges, it does not create the artifacts (the files say that is a separate Art skill).
  • Anyone without an existing eval setup or stored outputs: the files show it depends on eval suites, a configured target, and a stored results folder, so it is of little use on its own.
  • Quick one-off feedback on a single item: it is built for batch sessions and a persistent taste profile, which is heavier than needed.

Try asking

  • “calibrate social posts”
  • “calibrate 24 social posts”
  • “show me outputs to rate, then tell me how well the art judge matches me”
  • “update the taste profile from my calibration notes”
Step-by-step guide: how to use Calibrate Judge to TasteInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 9 original files, the MIT licence and an attribution note.

    Calibrate.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip Calibrate.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip Calibrate.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Run a small session first (for example 4 or 16 items) to see the flow before committing to larger batches.
  • Add a short note whenever a score matters, since notes are what the taste profile is distilled from.
  • Review the calibration report before accepting a new taste profile, and only promote a judge with your explicit approval, as the skill's own ideal state requires.

What's inside

  • Intents/AddTarget.md
  • Intents/Distill.md
  • Intents/Report.md
  • Intents/Run.md
  • Setup.md
  • SKILL.md
  • Tools/Calibrate.help.md
  • Tools/Calibrate.ts
  • Tools/ReviewPage.ts

Redistributed unmodified from danielmiessler/LifeOS under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.