Skip to content
Get packs

A/B Test Setup

Plans, designs and analyses A/B tests with sound statistics and clear hypotheses

Source Download skill
Repository
coreyhaines31/marketingskills
Stars
57,951
Licence
MIT
Files
4 files

What it does

Once installed, the agent acts like an experimentation expert when you want to compare two versions of something, such as a headline, button or page. It helps you write a clear hypothesis, pick one primary metric plus guardrail metrics, estimate how many visitors you need and how long to run the test, and avoid common mistakes like stopping early. It also explains statistical results (p-values, confidence intervals) in plain terms. The files are instructional guidance and reference tables; they do not set up any tool for you.

Great for

  • Marketers or product managers planning a website or app experiment
  • Developers implementing a feature flag or split test who need guidance on sample size and metrics
  • Anyone who wants to build a repeatable experimentation process, backlog or ICE-style prioritisation

Skip it if

  • Tracking setup or analytics instrumentation, which the skill says to hand off to a separate analytics skill (the skill does not do it itself)
  • Sites with very low traffic, where the sample-size tables show tests may take impractically long to reach valid results
  • Decisions that need a verified statistical audit: the guidance is general and the agent may still make calculation mistakes, so check numbers with the linked calculators

Try asking

  • “Help me design an A/B test for our pricing page headline and tell me how many visitors we need”
  • “We have two signup flow versions; how long should I run this test before deciding?”
  • “Write a hypothesis and guardrail metrics for testing a larger CTA button”
  • “Explain whether my test result with p = 0.04 means the new version is definitely better”
Step-by-step guide: how to use A/B Test SetupInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    ab-testing.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip ab-testing.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip ab-testing.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent your baseline conversion rate, weekly traffic and the smallest lift you care about so it can size the test properly.
  • Decide the primary metric and stopping rule before launching, and tell the agent so it can check your plan against the peeking warnings.
  • If you have product marketing context saved in .agents/product-marketing.md or .claude/product-marketing.md, the skill will read it first to avoid repeated questions.

What's inside

  • evals/evals.json
  • references/sample-size-guide.md
  • references/test-templates.md
  • SKILL.md

Redistributed unmodified from coreyhaines31/marketingskills under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.