Guide · Agent skill
How to use the A/B Test Analysis agent skill
Updated · written with AI from the project's own documentation and checked against our catalogue data
The A/B Test Analysis skill is an agent skill from the phuryn/pm-skills collection (pm-data-analytics) that tells an AI agent how to evaluate A/B test results with statistical rigor and turn them into a product decision. It is useful if you are a product manager checking whether an experiment reached significance, or an analyst who wants the same checklist applied every time. By the end of this guide you will know what the skill changes about the agent's behaviour, how to install it, how to ask for an analysis, and where its limits are.
A/B Test Analysisphuryn/pm-skills · MIT Download the skill Quick start
- 1Download the ZIP. Get the ab-test-analysis skill ZIP from this page.
- 2Unzip into a skills folder. Unzip into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder.
- 3Or upload in the Claude apps. Go to Settings → Capabilities → Skills and upload the ZIP.
- 4Gather the experiment details. Collect the hypothesis, variant, primary metric, guardrails, duration, traffic split and sample sizes or raw data file.
- 5Ask in plain language. Describe the result and ask whether it reached significance or whether to ship the variant.
- 6Review the summary table. Check the lift, p-values, confidence intervals and guardrail rows against your own numbers.
- 7Act on the recommendation. Use the Ship, Extend, Stop, Investigate or Don't ship label as a starting point and validate high-stakes decisions with an expert.
What is an agent skill and what does this one make the agent do?
An agent skill is a folder containing a SKILL.md file of instructions that an AI agent loads when a request matches the skill's description. This skill's description says it applies when you evaluate experiment results, check whether a test reached significance, interpret split test data, or decide whether to ship a variant.
Once loaded, the agent follows a fixed sequence. It first confirms the experiment's hypothesis, the change made in the variant, the primary metric, any guardrail metrics, how long the test ran and the traffic split. It then checks the setup for sample size, duration, sample ratio mismatch and novelty effects, and calculates conversion rates, relative lift, p-values and 95% confidence intervals.
- Checks whether the sample is large enough for the expected effect, flagging tests under 80% power.
- Checks whether the test ran for at least one to two full business cycles.
- Looks for sample ratio mismatch (SRM) in the randomization.
- Checks guardrail metrics such as revenue, engagement or page load time.
- Produces a summary table and one recommendation: Ship, Extend, Stop, Investigate or Don't ship.
How do you install the skill?
You can install the skill in either of two ways. Download the ZIP from the skill's page on this directory and unzip it into your personal skills folder, ~/.claude/skills/, or into a project's .claude/skills/ folder if you want it only for that project.
Alternatively, in the Claude apps go to Settings, then Capabilities, then Skills, and upload the ZIP there. The skill is distributed as a single SKILL.md file, so the folder you unzip should contain that file.
- Personal install: unzip into ~/.claude/skills/
- Project install: unzip into the project's .claude/skills/ folder
- Claude apps: Settings → Capabilities → Skills, then upload the ZIP
How do you trigger the skill with a natural request?
You do not need a special command. Describe what you have in plain language, and mention the experiment's result, significance or the ship decision. The skill's description lists the situations it is meant for, so requests that match those tend to load it.
Give the agent as much context as you can: the hypothesis, the variant, the primary metric, the guardrails, the dates and the traffic split. The skill's own instructions ask for these details, and an analysis without them will have gaps.
- "Did our new checkout test reach significance? Control converted 4.1% of 12,000 users, variant 4.6% of 12,000."
- "Here is an export of the pricing page test (CSV). Should we ship the variant to everyone?"
- "Check whether this A/B test had sample ratio mismatch before we read the results."
- "Interpret this split test: the primary metric is signup rate and page load time is our guardrail."
What do you need before you start?
You need the test results and enough context about the experiment. If you have raw data, supply it as CSV, Excel or an analytics export. The skill reads these files directly.
Python is needed only when you provide raw data, because the agent then writes and runs Python scripts for the calculations. If you only give summary numbers, the source does not require Python.
- Test data as CSV, Excel or analytics exports (optional, but needed for script-based calculations)
- Python, only if raw data is supplied
- The hypothesis, variant, primary metric, guardrail metrics, duration and traffic split
What does the output look like?
The skill asks the agent to produce a markdown summary with the hypothesis, duration and sample sizes, a table of control and variant values for the primary and guardrail metrics with lift, p-value and significance, and then the recommendation, reasoning and next steps. The agent is directed to save the result as markdown.
The recommendation follows a fixed mapping. A significant positive lift with no guardrail problems leads to Ship it. A significant positive lift with guardrail concerns leads to Investigate. A non-significant positive trend leads to Extend the test. A non-significant flat result leads to Stop the test. A significant negative lift leads to Don't ship.
## A/B Test Results: [Test Name]
**Hypothesis**: [What we expected]
**Duration**: [X days] | **Sample**: [N control / M variant]
| Metric | Control | Variant | Lift | p-value | Significant? |
|---|---|---|---|---|---|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |
**Recommendation**: [Ship / Extend / Stop / Investigate]
**Reasoning**: [Why]
**Next steps**: [What to do]What are the safety notes?
This skill runs scripts, because the agent generates and runs Python for calculations when raw data is provided. Review generated code before you rely on its output, especially if the data is sensitive. The skill does not use the network, according to its listing.
The recommendation is only as good as the inputs. The skill is not a substitute for a statistician when the stakes are high.
- Scripts: yes, Python is written and run when raw data is provided.
- Network: no.
- Review the generated code and the data you feed in before acting on results.
What people use it for
- Deciding whether a checkout button variant should roll out to all users after a two-week test.
- Checking whether a pricing page test has enough sample to detect the effect you care about.
- Confirming a test was not affected by sample ratio mismatch before reading its results.
- Weighing a conversion win against a drop in page load time or revenue as a guardrail.
- Writing a consistent markdown results summary to share with stakeholders.
Frequently asked questions
Is the A/B Test Analysis skill free and what is its licence?
The skill is listed under the MIT licence. The source material for this listing does not describe any pricing, so check the repository for current terms.
How do I install the ab-test-analysis skill in Claude?
Download the ZIP from this page and unzip it into ~/.claude/skills/ or a project's .claude/skills/ folder. Alternatively, upload the ZIP in the Claude apps under Settings → Capabilities → Skills.
Does the skill need Python?
Python is needed only if you provide raw data and the agent generates calculation scripts. If you only give summary numbers, Python is not listed as required.
What data formats can I give the agent?
The skill reads CSV, Excel and analytics exports directly. Give the agent the hypothesis and metric definitions alongside the file.
Can this skill analyze multivariate or sequential tests?
No. The skill is for A/B tests and does not explicitly cover multivariate or sequential designs.
When should I not rely on this skill?
Avoid it for very small samples or experiments without a pre-defined hypothesis, where results may mislead. Get a statistician to review decisions with major financial or safety consequences before acting on an AI-generated recommendation.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits product managers and analysts who want a consistent, checklist-driven review of A/B test results, including sample size, SRM and guardrail checks. It is less suitable for small or exploratory experiments, multivariate or sequential designs, or high-stakes decisions that need expert sign-off. Use it to structure your reasoning, not to replace a statistician.
Download the skill