CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 2 min read

What the example found

Sample size by effect size, A/B power at small samples, peeking inflation, canary stage detection, and paired-versus-unpaired power.

How to read the sample-size table: each row is a different baseline rate and target change to detect, at 80% power and a 5% two-sided significance level.

Baseline Change to detect Per arm
70% 5 points 1,251
70% 3 points 3,554
70% 2 points 8,080
90% 2 points 3,213
5% 2 points 2,213

What to notice: going from a 5-point to a 2-point target change at the same 70% baseline multiplies the needed sample more than sixfold (1,251 to 8,080). This is a direct consequence of the inverse-square relationship in the formula. It is the single number most worth remembering before agreeing to "just run a quick A/B test."

How to read the peeking table: each row is a different number of times the same A/A test (no real difference) was checked, comparing an uncorrected stopping rule against one that splits the error budget across looks.

Looks Stop at first p<0.05 Alpha split across looks
1 5% 5%
5 14% 4%
10 19% 3%
20 25% 2%

What to notice: the corrected column actually falls as looks increase, while the uncorrected column rises. The correction is not just capping the damage. It trades a little power for a false-positive rate that gets more conservative the more often you check. That is exactly the opposite of what peeking does uncorrected.

How to read the canary table: each row is a different size of regression, injected into the same three-stage rollout; the columns are the share of simulations stopped at each stage and the resulting extra bad outcomes served.

Regression Stopped at stage 1 / 2 / 3 Extra bad outcomes
Errors 2% to 6% 34% / 59% / 7% about 22
Errors 2% to 3% 13% / 12% / 74% about 40

What to notice: the large regression is overwhelmingly caught by stage 2 (34 + 59 = 93%), while the small one is mostly only caught at stage 3 (74%). That delay is exactly why the small one causes more total damage. It does so despite being, in percentage-point terms, the smaller of the two regressions.

{{svg:ladder|Five rungs from offline eval to full rollout, each with what it exposes and what it finds.}} {{svg:results|Peeking false positives, canary stopping stage, and paired versus unpaired power.}}