Part 9 of 12 · 2 min read
Mistakes that misread a benchmark score
Common mistakes, and what the example does not do.
Mistakes that misread a benchmark score
- Choosing a model on a leaderboard rank alone.
- Comparing rows with different scaffolds and budgets.
- Ignoring the benchmark's date and status.
- Quoting a benchmark you tuned on.
- Reading small gaps as meaningful.
- Using exact-match benchmarks to judge action-taking agents.
- No cost column.
What the example does not do
About this example What the example does not do
- No real benchmark data; difficulty and skill are drawn from simple distributions.
- Flaws are all-or-nothing; real flaws also let wrong solutions pass.
- Contamination is a fixed share of fully solved items.
- One agent run per task; repeated trials are in module 1.
- The facts about SWE-bench Verified are OpenAI's own account, as reported on its page.
- Optional real experiment: run two models on 100 GAIA validation questions with your harness, three times each, and compute the paired interval for their difference.
Check yourself
A model tops a public leaderboard. What does that tell you about your product?
Show answer
Only that it is worth shortlisting. Decide on your own eval set, and check the harness, budget and attempts before comparing rows.
For the curious Benchmarks retire
OpenAI said on 23 February 2026 that it had stopped reporting SWE-bench Verified; its audit of 138 problems found 59.4 percent had flawed tests or unclear descriptions (read 31 October 2026). Check the date and version of any benchmark you quote, and re-check current claims before relying on them.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Shortlist with public scores | Then test finalists on your own cases. |
| Read the harness | Tools, retries, time and cost budgets differ between rows. |
| Report cost next to score | A higher score at ten times the cost may not be better. |
| Check whether the benchmark is still current | Tasks get retired or superseded. |