CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 6 of 12 · 1 min read

Five questions to ask of a published benchmark

What a benchmark measures, how big it is, how clean, how contaminated, and which harness produced each row.

Benchmark Measures What it can miss (the assessment)
GAIA multi-step questions with web, files and tools; exact-match answers long, open-ended work; actions with side effects; cost and latency; answers may be findable after publication
SWE-bench family resolving real repository issues so that tests pass test flaws, contamination (OpenAI's audit), no measure of code review quality or maintainability
tau-bench (general knowledge) dialogue with tools under policy rules, repeated for consistency domain coverage; simulated users

The right-hand column is the judgment, partly from the sources above, not a published finding for each row.