CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that make test results meaningless

Common mistakes, and what the example does not do.

Mistakes that make test results meaningless

  • One run per case reported as the score.
  • No interval, and no suite size.
  • Comparing two agents on different cases or seeds.
  • Treating pass@k as reliability.
  • Assuming trials are independent when failures cluster by task.
  • Unfrozen environments: live web pages change between runs.
  • Reading differences below the noise floor as improvements.
  • Tuning prompts on the test set, until the score only measures the tuning (module 7 and 10).

What the example does not do

About this example What the example does not do
  • No real agent or model; the success draws are Bernoulli.
  • Independent tries given the task; real tries share causes (a bad tool, a confusing instruction).
  • A fixed suite in each experiment; the variation from choosing different tasks is only partly captured.
  • A perfect retry detector in the chain table.
  • Optional real experiment: run one agent 10 times on 30 tasks, then compute the suite score for each run, the interval, pass@5 and pass^5, and the share of tasks that are inconsistent.

Check yourself

A task passes 80 percent of the time per try. Your agent gets four tries. What is pass@4, and what is the chance all four tries pass?

Show answer

pass@4 is 0.998; all four pass is 0.41. The first says it can ever solve it; the second says you can rely on it.

For the curious Why a small suite crowns false winners

Two identical agents, one run each: on 20 tasks one looked at least 3 points better 84 percent of the time; on 300 tasks, 35 percent. The standard deviation of one suite score was 0.077 at 20 tasks and 0.021 at 300. Before you believe a gap, run the same version twice and keep that difference as your noise floor. These numbers come from a simulation with assumed task difficulty.

What you will use daily at work

Practice What it looks like
Report a score with its interval 80 percent on 10 cases is 49 to 94 percent; on 100 cases it is 71 to 87.
Run each case 3 to 5 times Repeats cut the noise and let you see unreliable cases.
Keep a noise baseline Run the same version twice; changes below that gap are noise.
Say how big the suite was Put the case count next to every score.