Guide · Agent skill
How to use the Verify Python Change Tests agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The agentic-verify skill is a set of instructions that tells Claude how to check whether a Python change is actually covered by tests. Instead of asking you to read the diff, it deliberately breaks the changed lines in small ways, confirms the tests catch each break, and writes or strengthens tests until they do. This guide explains what the skill does, how to install it, how to trigger it, and what it needs to run.
Verify Python Change Testssickn33/agentic-awesome-skills · MIT Download the skill Quick start
- 1Install the skill. Unzip the downloaded ZIP into ~/.claude/skills/ for personal use, or .claude/skills/ inside a project, or upload it under Settings → Capabilities → Skills.
- 2Check requirements. Confirm Python 3.12+, pytest and pytest-cov are in the project environment, pytest-randomly is installed, and uv is available.
- 3Make the change and its tests. Ask Claude to implement the Python change and write tests that check the behaviour a user depends on.
- 4Ask for verification. Request a check in plain language, such as whether the change is tested, and name the base revision if it is not HEAD.
- 5Run verify. The skill runs the pinned runner with verify against the base revision and reads the verdict.
- 6Close gaps. For gaps, the skill adds tests for surviving mutants or uncovered lines, then reruns verify, for up to three rounds.
- 7Read the report. Review the evidence: the verdict, mutants killed, new tests failing on the old code, and the path to report.md.
What is an agent skill, and what does this one do?
An agent skill is a folder with a SKILL.md file that gives an AI agent such as Claude extra instructions for a specific task. When the task matches the skill, Claude follows its steps.
This skill applies to Python projects that use pytest. After Claude writes or changes code, it defines the success criteria, writes tests that should fail before the change, and runs a mutation checker on the changed lines. The checker creates small wrong versions of the code (mutants). A mutant that a test catches is "killed". The skill keeps working until the surviving mutants are dealt with or explained.
- Defines what success looks like before coding or testing.
- Runs every mutant of the changed lines with a pinned runner.
- Adds or strengthens tests until they catch the mutants.
- Reports evidence: the verdict, mutants killed, and new tests that fail on the old code.
Install the skill in Claude
Download the ZIP from this page and unzip it. You can install it for yourself or for one project.
For all your projects, unzip the skill into ~/.claude/skills/. For a single project, unzip it into the project's .claude/skills/ folder. You can also upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
Prepare your project
The skill needs a Git repository with a base revision to compare against. Check that your project meets these requirements before you run it.
- Python 3.12 or newer.
- pytest and pytest-cov in the project environment.
- pytest-randomly, for the order check.
- uv, so that uvx can run the runner pinned from GitHub.
- A Git repository with a base revision such as HEAD or main.
Trigger the skill with natural requests
You do not need a special command. Ask Claude in plain language after it has finished a Python change. The skill also takes optional arguments: --base sets the git revision to compare with (HEAD by default, which checks uncommitted work; a branch such as main checks the whole branch), and --budget-minutes sets the time budget. The path to the project can also be given.
- "Is this change actually tested?"
- "Verify the Python change before we say it's done."
- "Check that the new code is covered by tests, comparing against main."
- To review a whole test suite instead, the skill's docs point to @agentic-test-audit.
Run the runner yourself
The skill calls the runner for you, but you can run the same commands. The project root is the directory pytest runs in, not a package or a single file. The runner works in a disposable copy and writes only to .test-suite-audit/ in your project. The last line of output is JSON with the verdict and report paths.
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV verify <project-root> --base <rev> [--budget-minutes N]
$AV replay --run <run-dir> --mutant <id> --all
$AV base-check <project-root> --base <rev> [tests...]Read the verdict and what to do next
The verdict tells you where the change stands. verified means every changed line runs under a test and every mutant of the changed lines was killed. gaps means some changed lines are not run by a test, some mutants survived, or the suite is not green and deterministic. incomplete means the time budget ran out or no mutant applied to the changed lines, so the mutants give no evidence. no change means nothing under the source packages changed since the base revision.
The skill allows up to three rounds of closing gaps. A surviving mutant has a patch file in mutants/ that shows the wrong version of your code the tests accept; the skill writes a test that passes on the real code and fails on that version.
- verified: go to the report and final checks.
- gaps: close each gap, then rerun verify.
- incomplete: rerun with a larger --budget-minutes, or rely on base-check and say so.
- no change: check the base revision.
Examples
Check uncommitted work with a 10-minute budget:
- Replay one surviving mutant with --all after adding a test, to confirm it is now killed (killed_now: true).
AV='uvx --from git+https://github.com/timini/agentic-verification@07608b3b9e4e359bc0b93b3eca94bb78d67e369e#subdirectory=runner agentic-verification'
$AV verify . --base HEAD --budget-minutes 10Safety notes: scripts and network
This skill runs scripts and uses the network. The runner is fetched from GitHub with uvx at the pinned commit shown above, and it runs your project's tests. The skill's rules forbid it from weakening, skipping or deleting existing tests to get a passing verdict, from retrying flaky tests until they pass, and from stashing, reverting or editing production files to see a test fail. Use base-check, which works in a copy. Review the commands before you let the agent run them in a project with sensitive data.
- The runner runs in a disposable copy and writes only to .test-suite-audit/.
- A runner verdict does not prove that code is bug-free.
What people use it for
- Checking an uncommitted bug fix in a pytest project before merging it.
- Confirming that a new feature in a Python library has tests that would notice a broken condition.
- Reviewing AI-written Python changes from a team before they are merged.
- Replaying a surviving mutant after adding a test to confirm the test now catches it.
Frequently asked questions
Is the agentic-verify skill free?
The skill is published under the MIT licence, as stated on this page. The runner is fetched from GitHub when the skill runs, and the source lists the license at the upstream repository.
What do I need to install agentic-verify?
You need to download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it in the Claude apps under Settings → Capabilities → Skills. The skill also needs Python 3.12+, pytest, pytest-cov, pytest-randomly and uv.
Does agentic-verify work with languages other than Python?
No. The skill targets Python and pytest only, so it is the wrong tool for other languages. For reviewing a whole test suite rather than one change, the skill's docs point to agentic-test-audit.
What does a mutant mean in this skill?
A mutant is a deliberately broken version of a changed line. If a test fails when it runs against the mutant, the mutant is killed, which shows the tests notice that change. A surviving mutant means no test caught it.
Why does verify report incomplete?
The verdict is incomplete when the time budget ended first or when no mutant applies to the changed lines, for example a decorated function. In either case the mutants give no evidence, so rerun with a larger --budget-minutes or rely on base-check and say so.
Does a verified result mean my code is bug-free?
No. The skill states that a runner verdict describes coverage and the mutants tested, and cannot prove that every possible defect would fail a test.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use agentic-verify if you write Python with pytest and want evidence that new code is actually tested before merging, especially for AI-written changes. Pick something else for non-Python projects, for reviewing a whole test suite (use agentic-test-audit), or where you need a guarantee of bug-free code.
Download the skill