Part 3 of 12 · 2 min read
Checking the result repeatedly is itself a source of false positives
Each time you check an experiment's result and could stop if it looks good, you're giving chance one more roll of the dice to fool you.
Think of flipping a coin and declaring it biased the moment you see any run of heads. If you check after every single flip, you are almost guaranteed to find some such run eventually. This happens even with a perfectly fair coin.
In plain words: peeking is checking results repeatedly and stopping at the first moment they look significant, and this inflates the false-positive rate well above the stated significance level, because each additional look is another independent chance for ordinary noise to cross the threshold by coincidence -- so a stopping rule has to be decided before the test runs, either a single fixed sample size or a sequential method that spends its error budget deliberately across looks.
A worked example. With no real difference between the two versions at all (an A/A test), stopping at the first time the p-value dips below 0.05 gives a false-positive rate of just 5% with a single look, as the significance level promises -- but 14% with 5 looks, 19% with 10 looks, and 25% with 20 looks. Splitting the same 5% error budget across the looks in advance (a Bonferroni correction) holds the false-positive rate down to 4%, 3%, and 2% respectively at those same look counts -- conservative, but nowhere near the uncorrected blowup.
The key lines. From code/mini-rollout/rollout.py:
# from code/mini-rollout/rollout.py
def peeking_false_positive(n_per_arm_total=2000, looks=10, correct=False, trials=1500, seed=2, p=0.5):
"""A/A test (no real difference). Stop at the first look whose |z| exceeds the threshold."""
rng = random.Random(seed)
zc = 1.959964
if correct:
from statistics import NormalDist
zc = NormalDist().inv_cdf(1 - 0.05 / (2 * looks))
Read it as: the function sets up a test where the two versions are identical (p=0.5 for both). So any "significant" result it finds is pure false alarm by construction. This is what makes the demonstration honest. There is no real effect anywhere in this simulation. Every stop it records before the correction is applied is, by design, a false positive and nothing else.
Peeking
Peeking is checking results repeatedly and stopping when significant, which inflates false positives.
Example: a false-positive rate climbing from 5% at one look to 25% at twenty looks, on a test where nothing is actually different, is peeking's entire cost made visible -- which is exactly why the module's design rules say to fix the sample size in advance rather than watch a dashboard and stop whenever it looks good.