Part 6 of 12 · 1 min read
Five questions to ask of a published benchmark
What a benchmark measures, how big it is, how clean, how contaminated, and which harness produced each row.
| Benchmark | Measures | What it can miss (the assessment) |
|---|---|---|
| GAIA | multi-step questions with web, files and tools; exact-match answers | long, open-ended work; actions with side effects; cost and latency; answers may be findable after publication |
| SWE-bench family | resolving real repository issues so that tests pass | test flaws, contamination (OpenAI's audit), no measure of code review quality or maintainability |
| tau-bench (general knowledge) | dialogue with tools under policy rules, repeated for consistency | domain coverage; simulated users |
The right-hand column is the judgment, partly from the sources above, not a published finding for each row.