Chapter 11 of 14 · Create · 5 min read
Testing, iterating and versioning a skill
A skill is software written in English, and like any software it can be wrong, regress after a change or behave differently from what you intended. The good news is that testing a skill doesn’t need a framework. It needs a handful of realistic requests, a clear idea of what good output looks like and the discipline to change one thing at a time. This chapter gives you a lightweight process that works for personal skills and scales up for team ones.
Three questions every test answers
| Question | What you check | Typical fix when it fails |
|---|---|---|
| Does it trigger? | The skill loads for requests it should handle, and not for others | Rewrite the description (see Chapter 9) |
| Does it follow the steps? | The agent does what SKILL.md says, in order, including the awkward cases | Make steps clearer, shorter or more explicit; move detail out |
| Is the result better? | Output with the skill beats output without it | Add a rule or example for the gap, or drop the skill |
The third question is the one people skip. Before investing in a skill, run the same tasks without it. Sometimes the model already does the job well, and the skill only adds cost. That’s a useful finding, not a failure.
Build a small test set
Collect five to ten realistic tasks. Use real inputs where you can: an actual branch, a real CSV, a genuine page you need to audit. Include at least one awkward case, such as missing data, an ambiguous request or a very large input.
tests/cases.md: a test set kept next to the skill
# pr-description test cases
## 1. Small bug fix
Branch: fix/null-email (2 commits)
Expect: one-line summary, "Why" mentions the crash, Testing lists the new test.
## 2. Large feature
Branch: feature/csv-export (14 commits, 22 files)
Expect: behaviour-level bullets (not a file list), Risks mentions memory.
## 3. No tests run
Branch: chore/rename-config
Expect: Testing says no tests were run. Must not invent testing.
## 4. Unclear reason
Branch: wip/misc
Expect: agent asks the user why before writing "Why".
## Should not trigger
- "Write a commit message for this"
- "Review this PR"Write the expected behaviour before you run the test. If you look at the output first, it’s too easy to decide that whatever came back is what you wanted.
Run, observe, adjust
- Start a fresh session for each test so earlier conversation doesn’t help or hinder.
- Watch what the agent reads and runs, not just the final output. Did it load the skill? Did it open the reference file when it should have?
- Score each case simply: pass, partial or fail, with a note on what went wrong.
- Change one thing in the skill, the smallest change that addresses the most common failure.
- Rerun the whole set, not just the failing case, to catch regressions.
Reading failures
- Skipped steps usually mean the body is too long or the step is buried. Move it up, number it, or make it a short checklist.
- Ignored rules often need a reason. “Never claim tests you didn’t run, because reviewers rely on this section” works better than a bare rule.
- Copied examples mean the example is too close to real tasks. Label it clearly or add a second, different example.
- Inconsistent output across runs suggests the step needs a template or a script.
- Over-eager behaviour, such as opening the pull request without being asked, needs an explicit “don’t” in the instructions.
Agents are not deterministic, so one good run proves little. Run important cases more than once, and be suspicious of a change that fixes one case while quietly breaking another.
Ask the agent to help
Agents are good critics of skills. After a test run, ask: “Which instructions in the skill were unclear or hard to follow?” or “What would you add so you’d get this right next time?” Treat the answer as suggestions to evaluate, not as edits to accept blindly. Skill-building skills can also automate parts of this, including running a test set with and without the skill and comparing the results.
Skill CreatorAgent skillCan run test cases and compare results with and without a skill.Agent skill · MIT
Verification Before CompletionAgent skillA skill that insists on evidence before claiming success.Agent skill · MIT
promptfooOpen sourceAn open-source tool for repeatable LLM evaluations.★ 26.7k · MITVersion it like code
Keep skills in Git, whether in your project’s .claude/skills/ folder or a dedicated skills repository. You get history, review and the ability to roll back a change that made things worse.
- Commit the skill and its test cases together, so anyone changing the skill can rerun the tests.
- Write commit messages that say what behaviour changed and why: “Ask for the reason when commits don’t explain it.”
- Review skill changes in pull requests like code. A one-line wording change can alter behaviour significantly.
- Tag releases if others depend on the skill, and keep a short changelog in the README.
- Some authors add a
versionfield to the frontmatter or a version line in the README. Hosts generally ignore unknown fields, but check before relying on it.
A typical change, from failure to commit
# 1. A test case failed: the agent invented testing on a branch with none.
# 2. Edit SKILL.md: strengthen the Testing rule and add a reason.
# 3. Rerun all cases in fresh sessions; all pass, nothing regressed.
git add .claude/skills/pr-description
git commit -m "pr-description: never claim testing that didn't happen
Case 3 failed: the agent listed tests on a rename-only branch.
Added the reason reviewers rely on this section."Know when to stop
A skill doesn’t need to be perfect, only reliably better than not having it. Once your test set passes consistently, stop editing. Each extra rule costs context and can conflict with others. Revisit when a new kind of failure turns up in real use, add it as a test case, and fix it then.
Key takeaways
- Test three things: it triggers, it follows the steps, the result is better.
- Keep a small set of realistic cases, with expected behaviour written first.
- Change one thing at a time and rerun everything in fresh sessions.
- Version skills and their tests in Git and review changes like code.
Put it into practice
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Written by the AI Tool Garage editors. Tool details link to their catalogue pages, which are refreshed weekly; spotted something out of date? Tell us.