Part 5 of 12 · 2 min read
What the example found
Ranking stability by benchmark size, detection power for real gaps, and the effects of flawed tasks and contamination.
How to read the ranking-stability table: each column is a different benchmark size; the value is how often the truly best of five close agents (skills 0.50-0.62) is ranked first, over 600 repeated trials.
| Tasks | 50 | 100 | 200 | 500 | 1,000 |
|---|---|---|---|---|---|
| Best agent ranked first | 54% | 63% | 76% | 88% | 96% |
What to notice: even at 1,000 tasks, the truly best agent still fails to rank first 4% of the time. Ranking stability improves steadily with size. But the table never promises certainty, only decreasing uncertainty.
How to read the detection-power table: two gap sizes, each at two task counts, using the paired bootstrap.
4-point gap: 22% detected at 100 tasks, 60% at 500. 2-point gap: 11% detected at 100 tasks, 26% at 500.
What to notice: doubling the gap size from 2 to 4 points more than doubles the detection rate at every task count. Detection power is far more sensitive to how large the real difference is than to a modest increase in sample size. That is why a benchmark comparison needs both a sizable gap and enough tasks to say anything with confidence.
How to read the flawed-tasks table: each row is a different share of broken, unpassable tasks in a 4,000-task benchmark, comparing agents of skill 0.62 and 0.50.
| Flawed share | Gap (points) | Ceiling |
|---|---|---|
| 0% | 12.5 | 100 |
| 20% | 9.7 | 80 |
| 40% | 7.8 | 60 |
What to notice: the gap shrinks almost exactly in proportion to the ceiling. Both drop by roughly 40% from the 0%-flawed row to the 40%-flawed row. Flawed tasks do not add random noise. They systematically compress every visible difference toward zero as the ceiling falls.
How to read the contamination table: each column is a different share of the test set a weaker agent has memorised; the stronger, uncontaminated agent's score is a fixed 64% throughout.
| Share memorised | 0% | 10% | 20% | 30% | 40% |
|---|---|---|---|---|---|
| Weaker agent's score | 45% | 50% | 54% | 61% | 67% |
What to notice: the weaker agent's score crosses the stronger agent's fixed 64% somewhere between the 30% and 40% memorised columns. A contamination share well within the range real audits have found is enough to invert a genuine skill ranking. The module's own "What the example does not do" section notes that the OpenAI SWE-bench Verified audit found most frontier models could reproduce verbatim problem details.
{{svg:questions|What is measured, how big, how clean, how contaminated, which harness.}} {{svg:results|Ranking stability, contamination and flawed-task effects.}}