Define Hypothesis and Metrics
1-2 daysState null and alternative hypotheses, primary metric, and minimum detectable effect.
Field context
This workflow is part of 3 niche fields
Complete guide for ab test analysis — step-by-step workflow, tools, checklist, and expert tips to get started.
State null and alternative hypotheses, primary metric, and minimum detectable effect.
Compute required sample size for desired power, alpha, and effect size.
Launch test, monitor for sample ratio mismatch, and collect data to planned duration.
Run significance tests, compute confidence intervals, and make ship/no-ship recommendation.
Calculate minimum sample size before launching the experiment.
Verify test has sufficient power to detect the minimum effect size.
Interpret p-values and confidence intervals after data collection.
Run chi-square test for categorical conversion metrics.
Key benchmarks for ab test analysis.
| Parameter | Typical | Notes |
|---|---|---|
| Alpha | 0.05 | False positive rate |
| Power | 0.80 | Detect real effect |
| MDE | 5-10% | Min detectable effect |
Checking results daily and stopping early when significant roughly doubles your false positive rate.
Include a buffer week beyond calculated duration to account for traffic variance.
Feature flag systems like LaunchDarkly or Optimizely amortize setup cost across many tests.