A/B Test Setup
Plan and check a split test: hypothesis, metrics, sample size, tracking and stopping rules
- Repository
- sickn33/agentic-awesome-skills
- Stars
- 49,023
- Licence
- MIT
- Files
- 1 file
What it does
When installed, the agent follows a structured procedure for designing an A/B (split) test on a product change. Before any variants are built, it makes the agent state a single hypothesis, a primary metric, the expected effect and a sample size estimate, and it checks that tracking works and that stopping rules are fixed in advance. It also warns against common mistakes such as stopping early because results look good or changing the test mid-run.
Great for
- Product or growth teams planning a website or app experiment who want a disciplined checklist
- Analysts who need a written, reviewable test plan with predeclared metrics and sample size
- Developers setting up event tracking for an experiment
Skip it if
- Low-traffic launches or qualitative research, where the skill itself says a randomized comparison is not appropriate and usability research is better
- Anyone expecting the skill to guarantee valid results; the file states the procedure cannot guarantee validity by itself
- Situations needing a statistics tool to compute results: the skill gives guidance and a sample-size example, not a verified calculator
Try asking
- “Design an A/B test for changing the checkout button text on our product page, including hypothesis and primary metric”
- “Estimate the sample size per variant needed to detect a 1 percentage point lift from a 10% baseline”
- “Review our tracking setup and confirm events are deduplicated and attributed to the right variant before launch”
- “Write a test record template for documenting results and the final decision”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 1 original files, the MIT licence and an attribution note.
ab-test-setup.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip ab-test-setup.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip ab-test-setup.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the agent your baseline conversion rate, expected traffic and the smallest effect worth detecting so the sample size estimate is realistic
- Lock the hypothesis and primary metric before asking for variant designs, since the skill treats these as gates
- Have a human verify the statistical formulas and the sample size output before relying on them for a business decision
What's inside
- SKILL.md
Redistributed unmodified from sickn33/agentic-awesome-skills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.