Part 5 of 12 · 2 min read
Small suites crown false winners between identical agents
Two copies of the exact same agent, run once each on a small suite, will often look like different agents.
Think of flipping the same coin twenty times in two separate batches. You would be surprised if the two batches did not show exactly 10 heads each. But the coin did not change between batches. The sample was just small enough to wobble.
In plain words: because a suite score is itself noisy, running two literally identical agents once each on a small suite will often make one look meaningfully better than the other purely by chance -- and the fix is either more tasks or more repeated trials per task, both of which shrink the chance of being fooled this way.
A worked example. Running the exact same agent twice (same code, same suite, no difference at all) on 20 tasks, the two runs differ by 3 or more points 84% of the time. On 300 tasks, that false-winner rate drops to 35% but doesn't vanish. Separately, detecting a genuine 5-point drop in quality is itself a detection problem: with 50 tasks and one run each, a true drop is correctly seen as worse only 70% of the time; running those same 50 tasks five times each raises that to 93%, and running 200 tasks five times each raises it to 99.9%.
The key lines. From code/mini-flaky/flaky.py:
# from code/mini-flaky/flaky.py
def false_winner(n_tasks, runs=3000, mean=0.7, margin=0.03, seed=2):
"""Two IDENTICAL agents, one run each: how often does one look better by at least `margin`?"""
rng = random.Random(seed)
ps = make_suite(n_tasks, mean, rng)
hits = 0
for _ in range(runs):
if abs(run_suite(ps, rng) - run_suite(ps, rng)) >= margin:
hits += 1
return hits / runs
Read it as: the function calls run_suite on the exact same ps (task probabilities) twice per simulated comparison. This represents the same agent evaluated twice. There is no agent difference anywhere in this code, so every "win" the function counts is, by construction, a false one. The number it returns directly measures how often noise alone would fool you.
Sampling error
Sampling error is the difference between a measured and a true value caused by finite samples.
Example: the 84%-of-the-time false-winner rate at 20 tasks is sampling error large enough to make a coin-flip-level decision look like a confident win -- which is exactly why a suite needs either more tasks, more repeats, or both before a score difference can be trusted.