Analyzes an A/B experiment to a defensible verdict - sample-size and minimum-detectable-effect math, two-proportion significance testing with confidence intervals, and checks for the traps that fake a win (peeking, sample ratio mismatch, multiple comparisons). Use when someone asks "is this test significant", "did variant B win", "how long should I run this experiment", "what sample size do I need", or shows experiment results and asks whether to ship. Do NOT use for logging and organizing ML training runs - use experiment-tracking instead; for causal questions without a randomized experiment use causal-inference; for general funnel or metric investigation use funnel-analysis or product-analytics.
Click to play with sound.
---
name: A/B Test Analyzer
description: Analyzes an A/B experiment to a defensible verdict - sample-size and minimum-detectable-effect math, two-proportion significance testing with confidence intervals, and checks for the traps that fake a win (peeking, sample ratio mismatch, multiple comparisons). Use when someone asks "is this test significant", "did variant B win", "how long should I run this experiment", "what sample size do I need", or shows experiment results and asks whether to ship. Do NOT use for logging and organizing ML training runs - use experiment-tracking instead; for causal questions without a randomized experiment use causal-inference; for general funnel or metric investigation use funnel-analysis or product-analytics.
---
# A/B Test Analyzer
Read experiments like a skeptic: most "wins" are noise until proven otherwise. The costly mistake this skill prevents is shipping on a peeked, underpowered, or mis-split test - a false positive that ships is worse than no test, because the team now defends a change that does nothing. Every verdict here comes with the interval, the checks that passed, and the checks that did not.
## Operating procedure
Order is mandatory: validity checks (Step 3) come before the significance math (Step 4), because a significant result on an invalid test is still invalid.
### Step 1: Gather inputs
Collect before computing anything; label unknowns as guesses.
1. The single primary metric and its type (conversion rate, revenue per user, retention).
2. The hypothesis and the minimum effect that would matter to the business - the smallest lift worth the cost of shipping and maintaining the change. If the requester has no number, help set one; without it "significant" is uninterpretable.
3. Per-arm sample sizes and metric values (successes and trials for proportions).
4. Whether the sample size and stopping rule were fixed before the test started, and whether anyone looked at results mid-flight.
5. Test duration and any events during it (marketing pushes, holidays, releases).
6. How many metrics and variants were compared.
### Step 2: Verify the test was designed to detect anything (power and MDE)… install to load the full skill