CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Flawed tasks compress real differences, and contamination can invert them

A benchmark with broken tasks, or one a model has secretly memorised, doesn't just add noise: it can make a worse agent look like the winner.

Think of a cooking competition where a quarter of the recipe cards have a typo that makes the dish impossible to complete correctly. The contest no longer measures who is the best cook. It measures who can still score points despite the broken cards. And if one contestant had secretly seen the recipe cards in advance, their advantage has nothing to do with cooking skill at all.

In plain words: saturation means scores approach the maximum so differences vanish, contamination means benchmark items or their solutions appear in a model's training data, and the ceiling is the highest achievable score, which sits below 100% once some tasks are simply flawed and unpassable by anyone -- flawed tasks compress the visible gap between a strong and a weak agent toward that lower ceiling, while contamination can reward memorization enough to flip the ranking entirely.

A worked example. Comparing a strong agent (skill 0.62) against a weaker one (skill 0.50) on a benchmark with no flawed tasks shows a 12.5-point gap, against a ceiling of 100%. With 40% of tasks flawed, the visible gap shrinks to 7.8 points and the ceiling drops to 60% -- the two agents look much closer than they really are, purely because so many tasks are unwinnable by either of them. Separately, a weaker agent (skill 0.45, clean score 45%) that has simply memorised 40% of the test set during training scores 67% -- actually beating a genuinely stronger, uncontaminated agent's 64%. The ranking is inverted, and nothing about the weaker agent's real ability changed.

The key lines. From code/mini-bench/bench.py:

# from code/mini-bench/bench.py
def contaminated_score(skill, memorised_share, n_tasks=2000, seed=4):
    """Items the agent saw in training are solved; the rest by skill."""
    rng = random.Random(seed)
    diffs = task_difficulties(n_tasks, rng)
    ok = 0
    for d in diffs:
        ok += True if rng.random() < memorised_share else (rng.random() < solve_prob(skill, d))
    return ok / n_tasks

Read it as: a memorised item is scored True unconditionally. It completely bypasses the solve_prob skill calculation that every other item goes through. This is the mechanism of contamination made explicit in code. A memorised fraction of the benchmark does not test skill at all. It tests whether the item was in the training data. The two get added together into one misleading final score.

Saturation, contamination, ceiling

Saturation is scores approaching the maximum, so differences vanish. Contamination is benchmark items or their solutions appearing in a model's training data. Ceiling is the highest achievable score, below 100 percent if some tasks are flawed.

Example: a memorised 40% share flipping a 19-point true skill gap (0.62 vs 0.45 skill levels) into a 3-point reversed lead for the weaker agent is contamination's real danger stated as one number -- the leaderboard doesn't just understate the gap, it can report the wrong winner entirely.