Part 10 of 12 · 1 min read
Quick notes
- An agent score is a measurement with an error bar. The model's samples, the task set, the environment and the grader all vary.
- Report pass@k and pass^k, not one run: pass@k for "can it ever", pass^k (all k tries pass) for "can I rely on it".
- Long trajectories compound: a step that works 95 percent of the time gives a 20-step run that works about a third of the time. One retry with perfect detection helps a lot; real detection is imperfect.
- Small suites produce false winners: two identical agents often look several points apart on 20 or 50 tasks.
- Intervals tell you how much to believe a number, and repeating trials per task helps detection as much as adding tasks.