Part 2 of 12 · 2 min read
An A/B test needs a sample size computed before it starts, not checked after
Asking "is this change better" without first asking "how many users would prove it" gets you a test that mostly measures noise.
Think of a poll that interviews five people to call an election. The sample is so small that almost any result it produces is just noise. It does not matter how the five people actually voted.
In plain words: an A/B test randomly splits users between versions to measure a metric difference, power is the probability of detecting a real difference of a given size, and sample size is the number of users needed per arm to reach a target power at a chosen significance level -- which means the sample size has to be computed from the smallest change you actually care about catching, before the test runs, or the result tells you almost nothing.
A worked example. For a 70% baseline, detecting a 3-point change at 80% power needs about 3,554 users per arm; detecting a smaller 2-point change needs 8,080 -- nearly 2.3 times as many users for a change that's only one-third smaller. Running the same 3-point-change test with only 500 users per arm, far below the needed 3,554, gives a measured power of just 18%: the test would correctly detect the real change only about 1 time in 5, and the other 4 times would report "no difference" even though one truly existed.
The key lines. From code/mini-rollout/rollout.py:
# from code/mini-rollout/rollout.py
def n_per_arm(p, delta, alpha=0.05, power=0.8):
"""Two-sided two-proportion test, equal arms, normal approximation."""
za = {0.05: 1.959964, 0.01: 2.575829}[alpha]
zb = {0.8: 0.841621, 0.9: 1.281552}[power]
p2 = p + delta
pbar = (p + p2) / 2
num = (za * math.sqrt(2 * pbar * (1 - pbar)) + zb * math.sqrt(p * (1 - p) + p2 * (1 - p2))) ** 2
return math.ceil(num / delta ** 2)
Read it as: the formula's final division is by delta ** 2, the detectable change squared. This single line is why halving the effect you want to catch roughly quadruples the users you need. The numbers 8,080 and 3,554 are not quite 4x apart because the baseline shifted too. But the squared denominator dominates. It is the one fact worth memorizing from the whole formula.
A/B test, power, sample size
An A/B test randomly splits users between versions to measure a metric difference. Power is the probability of detecting a real difference of a given size. Sample size is the users needed per arm for a target power at a significance level.
Example: 500 users per arm producing only 18% power for a 3-point change is a test that's essentially broken before it starts -- not because the change isn't real, but because the sample size was never checked against the formula that n_per_arm runs.