Part 8 of 12 · 2 min read
Answering benchmark questions in an interview
How would you use public benchmarks when choosing a model for an agent?
"To shortlist, not to decide. I check what the benchmark measures against my task, its size and noise, whether it is saturated or contaminated, and whether the rows used the same harness and budget. Then I evaluate the finalists on my own traces-based set with intervals and cost. A recent example is SWE-bench Verified, which OpenAI stopped reporting after finding flawed tests and contamination." Follow-up: "What is wrong with a leaderboard rank?" It hides error bars (on 100 tasks a 4-point gap is detected about a fifth of the time in the example's simulation), harness differences and cost.
What does GAIA measure?
"Real-world assistant questions, 466 of them, that need reasoning, browsing, multi-modality and tool use; they are simple for humans, who scored 92 percent in the paper, and were hard for the model with plugins, 15 percent. It scores final answers by exact match, so it says little about side-effecting actions, cost or long-running work." Follow-up: "Why do you use it in your own project?" As an agent gym: it is a convenient set of multi-step tasks with answers (Build an AI Agent from Scratch), but it is not a proxy for my product.
What is benchmark contamination and how does it matter?
"The model saw the test data or solutions during training, so the score measures memory. In the example's simulation a weaker agent that had memorised 40 percent of the set outscored a stronger clean agent, 67 to 64." Follow-up: "How would you check for it?" A verbatim-recall probe, held-out or freshly written tasks, and reporting results on a clean subset separately.
Two agents differ by 3 points on a 300-task benchmark. Which is better?
"I cannot tell from that alone. With a 3-point gap and 300 tasks the detection power is below one half in the example's simulation, and a paired comparison on the same tasks with repeated runs is needed. I also check harness and cost." Follow-up: "What would convince you?" A paired test whose interval excludes zero on a task set that resembles mine.