Guide · Agent skill
How to use the Experiment Integrity Audit agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The experiment-audit skill is an agent skill for Claude Code and compatible agent runtimes that checks your experiment results for integrity problems before you write them up. It is aimed at researchers, teams running AI agents on experiments, and students reviewing their own evaluation code. After this guide you will know what the skill asks the agent to do, how to install it, how to trigger it with plain requests, and what it needs to run.
Experiment Integrity Auditwanshuiyin/Auto-claude-code-research-in-sleep · MIT Download the skill Quick start
- 1Download the ZIP. Download the ZIP for skills/experiment-audit from the directory page. It contains SKILL.md.
- 2Unzip into a skills folder. Unzip into ~/.claude/skills/ for all projects, or into the project's .claude/skills/ for one project.
- 3Or upload in the Claude apps. In the Claude apps, open Settings → Capabilities → Skills and upload the ZIP.
- 4Install the reviewer MCP server. Make sure the Codex MCP server is installed, since it is the default reviewer. Choose Oracle or Manual Review only if you select them.
- 5Finish experiments and collect paths. Wait until your experiments are complete and results files are written. The agent then collects paths only, without summarizing them.
- 6Ask for an audit. Say something like 'audit results in this experiment folder' or 'check experiment integrity' and give the directory path.
- 7Review the findings. Read each finding the reviewer returns, check the cited file, and correct your claims before writing them up.
What the experiment-audit skill does
The skill runs after experiments finish and before results go into a paper or report. The agent gathers file paths for evaluation scripts, result files, the experiment tracker, paper claims and config files. It does not judge the work itself. It hands those paths to a separate reviewer model, which by default is Codex connected through an MCP server, and the reviewer reads the code and looks for integrity problems.
The checklist in the skill targets specific failure modes: ground truth that was generated from model outputs without being labelled as a proxy, metrics divided by the model's own maximum or mean, claims that point to result files or metric keys that do not exist, and small pilot runs described as comprehensive evaluations.
- Ground truth provenance: is the reference loaded from the dataset or derived from model outputs?
- Score normalization: is any metric divided by statistics of the model's own predictions?
- Result file existence: do the referenced files, metric keys and numbers actually match?
- Tracker status: is the experiment marked DONE rather than TODO or IN_PROGRESS?
How to install the skill
Download the ZIP for skills/experiment-audit from the directory page. The package contains a single file, SKILL.md. Unzip it so the folder sits in your skills directory. You can use either a personal skills folder or a project-specific one.
You can also upload the skill through the Claude apps. Open Settings, go to Capabilities, then Skills, and upload the package there.
- Personal install: unzip into ~/.claude/skills/ so the skill is available in all your projects.
- Project install: unzip into the project's .claude/skills/ folder so it applies only to that repository.
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP.
How to trigger the skill with natural requests
You do not need a special command. The skill's description lists phrases that should activate it, and you can also ask for it after experiments complete. The skill also accepts an optional argument, an experiment directory or results path, which the skill's argument hint describes.
Good requests name the experiment and say what you want checked before you write claims.
- '审计实验' (audit the experiment)
- 'check experiment integrity'
- 'audit results' for the folder you just ran
- '实验诚实度' (experimental honesty check)
- 'Experiments are done. Check integrity before I write the results section.'
What you need before running it
The skill needs Claude Code or a compatible agent runtime. The default reviewer is the Codex MCP server. You can instead select Oracle MCP with the option '— reviewer: oracle-pro', or Manual Review MCP with '— reviewer: manual'. Other reviewer servers listed in the skill's tool permissions (antigravity, grok, claude-review) are used only if you choose them.
If the selected reviewer MCP server is not installed, the skill stops and prints an install command rather than falling back to a weaker check. Plan on that before your first run.
- Claude Code or a compatible agent runtime
- Codex MCP server (default reviewer)
- Oracle MCP or Manual Review MCP, only if selected
- antigravity, grok or claude-review MCP servers, only if chosen
Safety notes: scripts and network access
The skill runs scripts and uses the network. Its tool permissions include Bash, file reads and writes, and calls to the reviewer MCP servers. Read the SKILL.md file before you install it, and check which reviewer you have selected, since that determines which external service receives your file paths and prompt.
Do not wrap this skill in /loop, /schedule or CronCreate. The skill issues a verdict on integrity, and re-running it on a timer adds no new evidence. Run it once, after the experiments are finished.
- Inspect SKILL.md before installing.
- Know which reviewer backend is active.
- Run the audit once after experiments finish; do not schedule it repeatedly.
Tips for a useful audit
Point the skill at the directory that holds the evaluation code, results and paper claims together, so the reviewer can cross-check them. Keep the experiment tracker up to date, because the reviewer checks whether tracked experiments are DONE. Treat the verdict as a second opinion: the audit is an AI judgment and can miss problems or raise false alarms, so verify flagged items yourself.
- Give the skill one experiment directory or results path at a time.
- Keep EXPERIMENT_TRACKER.md and paper claim files current.
- Read the reviewer's cited file and line for each finding before changing your claims.
What people use it for
- Checking a paper's results section for metrics that do not match the saved result files before submission.
- Reviewing an AI agent's evaluation script for reference data that was generated from the model's own outputs.
- Students reviewing their own evaluation code for normalization by the model's best score before a report is due.
- Confirming that an experiment marked TODO or IN_PROGRESS in the tracker is not cited as a finished result.
Frequently asked questions
Is the experiment-audit skill free?
The skill is published under the MIT licence, as listed on its directory page. The page does not list prices, and the reviewer you choose (Codex, Oracle or Manual Review) may have its own requirements that you should check separately.
What do I need to run the experiment-audit skill?
You need Claude Code or a compatible agent runtime and the Codex MCP server, which is the default reviewer. If you select Oracle MCP or Manual Review MCP, you need that server instead. The skill stops rather than running a weaker check when the selected reviewer is missing.
How do I install the experiment-audit skill in Claude Code?
Download the ZIP from the directory page and unzip it into ~/.claude/skills/ for personal use, or into a project's .claude/skills/ folder. In the Claude apps you can instead upload the ZIP under Settings → Capabilities → Skills.
Does the experiment-audit skill guarantee my results are correct?
No. The audit is an AI judgment and can miss problems or raise false alarms. It is a second opinion to check before you claim results, not a guaranteed correct verdict.
How does experiment-audit compare with asking the agent to review its own work?
The skill is designed so the executor agent only gathers file paths and does not judge its own work. A separate reviewer model reads the code and makes the integrity judgment.
Why did the experiment-audit skill stop without giving a verdict?
The most likely cause is that the reviewer MCP server you selected is not installed. The skill prints an install command and stops instead of falling back to Codex or another weaker check.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
Use experiment-audit if you want an adversarial second look at evaluation code and paper claims before you publish, and you are comfortable setting up an MCP reviewer. Pick something else if you only need help with code that has no experiments, or if you need a guaranteed correct verdict, since this skill is an AI judgment that can miss problems.
Download the skill