Part 3 of 12 · 2 min read
Detecting a real gap between two agents needs more tasks than you'd think
A genuine 4-point quality difference between two agents can easily look like noise on a benchmark of ordinary size.
Think of trying to tell whether one brand of lightbulb genuinely lasts a little longer than another by testing just a handful of each. A real but modest difference in average lifespan can easily be swamped by the ordinary bulb-to-bulb variation within each brand.
In plain words: a paired bootstrap resamples tasks while keeping both agents' results for each task, to see how often the sign of the difference changes, and power is the chance of actually detecting a real difference when one truly exists -- and because detecting a modest gap reliably requires a specific, computable number of tasks, a benchmark comparison with too few tasks can simply lack the power to tell two agents apart even when one really is better.
A worked example. For a true 4-point gap between two agents, a paired-bootstrap test detects it (at a conventional significance level) only 22% of the time on 100 tasks -- worse than a coin flip's worth of confidence. On 500 tasks, that detection rate rises to 60%. For a smaller, more realistic 2-point gap, detection is just 11% at 100 tasks and only 26% at 500 -- meaning even a fairly large benchmark will usually fail to confirm a real but modest quality difference exists at all.
The key lines. From code/mini-bench/bench.py:
# from code/mini-bench/bench.py
def paired_bootstrap_p(a, b, rng, reps=400):
"""Share of bootstrap resamples (of tasks) in which agent A does NOT beat B; small = A reliably better."""
n = len(a)
worse = 0
for _ in range(reps):
idx = [rng.randrange(n) for _ in range(n)]
da = sum(a[i] for i in idx) - sum(b[i] for i in idx)
worse += da <= 0
return worse / reps
Read it as: the resampling (idx) always draws the same task index for both a[i] and b[i] in each resample. This pairs the two agents' results on identical resampled tasks. The alternative is resampling each agent's results independently. Pairing removes task-to-task difficulty variation from the comparison. It leaves only the genuine difference between the two agents.
Paired bootstrap, power
A paired bootstrap resamples tasks, keeping both agents' result for each task, to see how often the difference changes sign. Power is the chance of detecting a real difference.
Example: a true 2-point gap being detected only 26% of the time even at 500 tasks is exactly why "two agents differ by a few points on a few hundred tasks" is not, on its own, evidence that one is actually better -- the test simply doesn't have enough power at that gap size and task count to tell you much either way.