Experiment Integrity Audit
Has an independent AI reviewer check your experiment results for fraud, fake ground truth and overclaiming.
- Stars
- 18,482
- Licence
- MIT
- Files
- 1 file
What it does
When you finish running experiments, this skill makes the agent collect the evaluation scripts, result files, and paper claims, then hand them to a separate reviewer model (by default Codex, via an MCP connection) that reads the code and looks for integrity problems. It checks for things like made-up reference data, metrics divided by the model's own best score to look better, numbers that come from files or functions that don't exist, and small pilot tests presented as comprehensive evaluations. The executor agent is meant to only gather file paths, not judge its own work.
Great for
- Researchers who want a second opinion before claiming results in a paper or report
- Teams using an AI agent to run experiments who want a check against overclaiming
- Students reviewing their own evaluation code before submission
Skip it if
- Simple one-off tasks or code without experiments, since the checklist is specific to evaluation integrity
- Situations where you need a guaranteed correct verdict: the audit is an AI judgment and can miss problems or raise false alarms
- Setups without the required reviewer MCP servers installed, because the skill stops rather than falling back to a weaker check
Try asking
- “Audit experiment integrity for the results in results/ before I write the paper claims”
- “Check experiment integrity of my eval pipeline in this project”
- “Audit results for the new benchmark and tell me if any claims are overstated”
- “实验诚实度检查一下这次的结果”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
experiment-audit.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip experiment-audit.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip experiment-audit.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Point it at a specific experiment directory or results path to keep the audit focused
- Run it once after experiments finish, not on a repeating timer, since the skill is designed as a one-time verdict
- Check which reviewer backend is used and that its model identity is reported, since cross-model independence depends on it
What's inside
- SKILL.md
Redistributed unmodified from wanshuiyin/Auto-claude-code-research-in-sleep under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.