Part 6 of 12 · 2 min read
What the example found
Sample size by effect size, A/B power at small samples, peeking inflation, canary stage detection, and paired-versus-unpaired power.
How to read the sample-size table: each row is a different baseline rate and target change to detect, at 80% power and a 5% two-sided significance level.
| Baseline | Change to detect | Per arm |
|---|---|---|
| 70% | 5 points | 1,251 |
| 70% | 3 points | 3,554 |
| 70% | 2 points | 8,080 |
| 90% | 2 points | 3,213 |
| 5% | 2 points | 2,213 |
What to notice: going from a 5-point to a 2-point target change at the same 70% baseline multiplies the needed sample more than sixfold (1,251 to 8,080). This is a direct consequence of the inverse-square relationship in the formula. It is the single number most worth remembering before agreeing to "just run a quick A/B test."
How to read the peeking table: each row is a different number of times the same A/A test (no real difference) was checked, comparing an uncorrected stopping rule against one that splits the error budget across looks.
| Looks | Stop at first p<0.05 | Alpha split across looks |
|---|---|---|
| 1 | 5% | 5% |
| 5 | 14% | 4% |
| 10 | 19% | 3% |
| 20 | 25% | 2% |
What to notice: the corrected column actually falls as looks increase, while the uncorrected column rises. The correction is not just capping the damage. It trades a little power for a false-positive rate that gets more conservative the more often you check. That is exactly the opposite of what peeking does uncorrected.
How to read the canary table: each row is a different size of regression, injected into the same three-stage rollout; the columns are the share of simulations stopped at each stage and the resulting extra bad outcomes served.
| Regression | Stopped at stage 1 / 2 / 3 | Extra bad outcomes |
|---|---|---|
| Errors 2% to 6% | 34% / 59% / 7% | about 22 |
| Errors 2% to 3% | 13% / 12% / 74% | about 40 |
What to notice: the large regression is overwhelmingly caught by stage 2 (34 + 59 = 93%), while the small one is mostly only caught at stage 3 (74%). That delay is exactly why the small one causes more total damage. It does so despite being, in percentage-point terms, the smaller of the two regressions.
{{svg:ladder|Five rungs from offline eval to full rollout, each with what it exposes and what it finds.}} {{svg:results|Peeking false positives, canary stopping stage, and paired versus unpaired power.}}