CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • An agent score is a measurement with an error bar. The model's samples, the task set, the environment and the grader all vary.
  • Report pass@k and pass^k, not one run: pass@k for "can it ever", pass^k (all k tries pass) for "can I rely on it".
  • Long trajectories compound: a step that works 95 percent of the time gives a 20-step run that works about a third of the time. One retry with perfect detection helps a lot; real detection is imperfect.
  • Small suites produce false winners: two identical agents often look several points apart on 20 or 50 tasks.
  • Intervals tell you how much to believe a number, and repeating trials per task helps detection as much as adding tasks.