Part 9 of 12 · 2 min read
Mistakes that make test results meaningless
Common mistakes, and what the example does not do.
Mistakes that make test results meaningless
- One run per case reported as the score.
- No interval, and no suite size.
- Comparing two agents on different cases or seeds.
- Treating pass@k as reliability.
- Assuming trials are independent when failures cluster by task.
- Unfrozen environments: live web pages change between runs.
- Reading differences below the noise floor as improvements.
- Tuning prompts on the test set, until the score only measures the tuning (module 7 and 10).
What the example does not do
About this example What the example does not do
- No real agent or model; the success draws are Bernoulli.
- Independent tries given the task; real tries share causes (a bad tool, a confusing instruction).
- A fixed suite in each experiment; the variation from choosing different tasks is only partly captured.
- A perfect retry detector in the chain table.
- Optional real experiment: run one agent 10 times on 30 tasks, then compute the suite score for each run, the interval, pass@5 and pass^5, and the share of tasks that are inconsistent.
Check yourself
A task passes 80 percent of the time per try. Your agent gets four tries. What is pass@4, and what is the chance all four tries pass?
Show answer
pass@4 is 0.998; all four pass is 0.41. The first says it can ever solve it; the second says you can rely on it.
For the curious Why a small suite crowns false winners
Two identical agents, one run each: on 20 tasks one looked at least 3 points better 84 percent of the time; on 300 tasks, 35 percent. The standard deviation of one suite score was 0.077 at 20 tasks and 0.021 at 300. Before you believe a gap, run the same version twice and keep that difference as your noise floor. These numbers come from a simulation with assumed task difficulty.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Report a score with its interval | 80 percent on 10 cases is 49 to 94 percent; on 100 cases it is 71 to 87. |
| Run each case 3 to 5 times | Repeats cut the noise and let you see unreliable cases. |
| Keep a noise baseline | Run the same version twice; changes below that gap are noise. |
| Say how big the suite was | Put the case count next to every score. |