Part 3 of 12 · 2 min read
Seeds and tuning budget can manufacture a gap with no real improvement at all
Reporting the best run, or giving the new method more tuning attempts than the baseline, creates an advantage that has nothing to do with the method itself.
Think of two identical runners racing the same course five times each. Only the faster of the two runners' five attempts gets reported. Of course that runner "wins." Both are equally fast on average.
In plain words: if a method's score varies from run to run (different random seeds) or across tuning attempts, reporting only the best run — or giving the proposed method more tuning tries than the baseline — creates an apparent improvement that vanishes the moment both sides get equal treatment.
A worked example. With a 1-point run-to-run spread, two genuinely equal methods that each report just one run will show the "new" one looking at least 1 point better 24 percent of the time; averaging 10 runs each cuts that to just 1 percent. Separately, if a method's score varies by 1.5 points across different configurations, a method that reports the best of 20 tuning tries beats an untuned single-try baseline of the same underlying quality by 2.81 points on average — purely from trying more configurations. Give both sides 20 tries each, and that gap drops to exactly 0.00.
The key lines. From code/mini-papers/stats.py:
# from code/mini-papers/stats.py
def tuning_gap(k_new, k_base=1, sigma_h=0.015, trials=40000, seed=0):
"""Equal methods, different tuning budgets. Each configuration scores N(0, sigma_h) around the same mean;
a method reports its best configuration. Returns the mean reported gap (new - base), in score units."""
rng = np.random.default_rng(seed)
new = rng.normal(0, sigma_h, (trials, k_new)).max(axis=1)
base = rng.normal(0, sigma_h, (trials, k_base)).max(axis=1)
return float(np.mean(new - base))
Read it as: new and base are drawn from the exact same distribution N(0, sigma_h). They differ only in how many tries (k_new vs. k_base) each gets before taking the max. The "gap" this function returns is entirely an artifact of unequal tuning budgets. Nothing about either method's real quality is baked in.
Ablation
An ablation removes or changes one component of a system to see how much of the reported result that component actually explains — it's the tool for checking whether a claimed gain survives once confounds like tuning budget are held equal.
Example: the 2.81-point gap above is exactly the kind of thing an ablation should catch — rerunning the comparison with equal tuning budgets for both methods (as the "20 against 20" row does, landing at 0.00) is a form of ablation applied to the experimental setup itself, not the model.