Experimental Design Best Practices
Guides the agent to plan ML experiments with proper baselines, seeds, ablations and controls.
- Repository
- aiming-lab/AutoResearchClaw
- Stars
- 14,802
- Licence
- MIT
- Files
- 1 file
What it does
This is a short set of written guidelines that tells an AI agent how to plan machine learning experiments the way careful researchers do. Once installed, when the agent is asked to design a test or comparison, it should include several reference methods to compare against, run each experiment with multiple random seeds, report averages with variation, remove one part of a system at a time to see what matters, change only one thing per comparison, keep test data separate from training data, and also record run time and memory use.
Great for
- Researchers or students planning a machine learning study who want a checklist for fair comparisons
- Engineers preparing benchmark results that need to be reproducible
- Teams using the AutoResearchClaw research pipeline, where this skill is applied at experiment-design stages
Skip it if
- Non-ML experiments or lab work, since the guidance is specific to machine learning benchmarks (the rules would not transfer cleanly)
- Quick exploratory coding where rigorous multi-seed evaluation would be unnecessary overhead
- Anyone expecting the skill to run experiments or check results: it contains only text guidance and does not verify the agent's actual work
Try asking
- “Design an ablation study for my transformer model with the baselines and seeds I should use”
- “Plan a controlled experiment comparing two data augmentation methods”
- “What baselines should I include when benchmarking my new image classifier?”
- “Help me set up reproducible experiments with proper train/val/test splits”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
experimental-design.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip experimental-design.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip experimental-design.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the agent your specific model, dataset and hypothesis so it can apply the baseline and ablation rules concretely
- Ask it to justify choices against the rules, then check the seed count and baselines yourself before spending compute
- Note that the guidance says 'minimum 3 seeds' and uses strong wording like ALWAYS; adapt it to your compute budget and field norms
What's inside
- SKILL.md
Redistributed unmodified from aiming-lab/AutoResearchClaw under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.