CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 5 of 12 · 2 min read

What the example found

Ranking stability by benchmark size, detection power for real gaps, and the effects of flawed tasks and contamination.

How to read the ranking-stability table: each column is a different benchmark size; the value is how often the truly best of five close agents (skills 0.50-0.62) is ranked first, over 600 repeated trials.

Tasks 50 100 200 500 1,000
Best agent ranked first 54% 63% 76% 88% 96%

What to notice: even at 1,000 tasks, the truly best agent still fails to rank first 4% of the time. Ranking stability improves steadily with size. But the table never promises certainty, only decreasing uncertainty.

How to read the detection-power table: two gap sizes, each at two task counts, using the paired bootstrap.

4-point gap: 22% detected at 100 tasks, 60% at 500. 2-point gap: 11% detected at 100 tasks, 26% at 500.

What to notice: doubling the gap size from 2 to 4 points more than doubles the detection rate at every task count. Detection power is far more sensitive to how large the real difference is than to a modest increase in sample size. That is why a benchmark comparison needs both a sizable gap and enough tasks to say anything with confidence.

How to read the flawed-tasks table: each row is a different share of broken, unpassable tasks in a 4,000-task benchmark, comparing agents of skill 0.62 and 0.50.

Flawed share Gap (points) Ceiling
0% 12.5 100
20% 9.7 80
40% 7.8 60

What to notice: the gap shrinks almost exactly in proportion to the ceiling. Both drop by roughly 40% from the 0%-flawed row to the 40%-flawed row. Flawed tasks do not add random noise. They systematically compress every visible difference toward zero as the ceiling falls.

How to read the contamination table: each column is a different share of the test set a weaker agent has memorised; the stronger, uncontaminated agent's score is a fixed 64% throughout.

Share memorised 0% 10% 20% 30% 40%
Weaker agent's score 45% 50% 54% 61% 67%

What to notice: the weaker agent's score crosses the stronger agent's fixed 64% somewhere between the 30% and 40% memorised columns. A contamination share well within the range real audits have found is enough to invert a genuine skill ranking. The module's own "What the example does not do" section notes that the OpenAI SWE-bench Verified audit found most frontier models could reproduce verbatim problem details.

{{svg:questions|What is measured, how big, how clean, how contaminated, which harness.}} {{svg:results|Ranking stability, contamination and flawed-task effects.}}