Covers: Definitions; Where the noise comes from and more.
Definitions
Non-determinism, trajectory, trial
Non-determinism: the same input can give different outputs.
Trajectory: the sequence of reasoning, tool calls and observations in one run.
Trial: one complete run of the agent on one task.
pass@k, pass^k, consistency
pass@k: the share of tasks solved by at least one of k tries.
pass^k (the notation used by tau-bench): the share solved by all k tries; the probability of success per try raised to the power k when tries are independent.
Consistency: giving the same correct result across repeated tries.
Confidence interval, Wilson interval, sampling error
Confidence interval: a range that would contain the true rate in about 95 percent of repeated experiments.
Wilson interval: an interval for a proportion that behaves well for small samples and extreme rates.
Sampling error: the difference between a measured and a true value caused by finite samples.
Where the noise comes from
The model, the task set and the environment each add noise; the grader adds a fourth.
Results
Consistency of a task with per-try success 0.8 (formulas):
k
pass@k
pass^k
1
0.80
0.80
2
0.96
0.64
4
0.998
0.41
8
1.000
0.17
Compounding over steps (all steps must work):
Steps
95% per step
99% per step
95% with one retry
1
0.95
0.99
0.997
5
0.77
0.95
0.988
10
0.60
0.90
0.975
20
0.36
0.82
0.951
50
0.08
0.61
0.882
Two identical agents look different on small suites; per-step reliability compounds over long trajectories.
Interval for a measured 80 percent: n=10: 0.49 to 0.94; n=30: 0.63 to 0.90; n=100: 0.71 to 0.87; n=300: 0.75 to 0.84; n=1000: 0.77 to 0.82.
Detecting a true 5-point drop (share of comparisons in which the worse agent scored lower): 50 tasks, one run: 0.70; 50 tasks, five runs: 0.93; 200 tasks, one run: 0.90; 200 tasks, five runs: 0.999.
Compare agents with intervals, not point scores. A 3-point gap on 20 tasks is within the noise of identical agents.
More tasks beat more repeats for the task-sampling error, and repeats beat tasks for sampling noise within a task; you usually need both.
Consistency is a separate property. A task at 0.8 passes once and fails often when you need every try to pass.
Step count is a design lever: shorter trajectories, retries with verification, and checkpoints (Level 8) raise end-to-end success more than a slightly better model.
Design rules
Run each case several times (3 to 5 at the start) and report the mean and an interval.
Fix what you can: seeds where the API allows, recorded tool responses, a frozen environment; accept what you cannot fix and measure it.
Report two numbers when it matters: pass@k (capability) and pass^k (reliability).
State the suite size next to every score.
Compare paired: run both versions on the same cases (module 7 uses this).
Keep a noise baseline: run the same version twice and record the difference; changes below it are noise.
Interview questions, with the follow-ups
How do you test an agent whose output changes on every run?
"I treat the score as a measurement. I run each case several times, report the mean with an interval, and report both pass@k and pass^k, because capability and reliability differ: for a 0.8 task, pass@4 is 0.998 but all four passing is 0.41. I fix the environment where I can, record tool outputs, and keep the suite big enough: in my simulation, two identical agents on 20 tasks looked 3 points apart 84 percent of the time." Follow-up: "How many cases?" Enough for the interval to be narrower than the effect I care about; for a proportion near 0.8 that is about 246 independent trials for plus or minus 5 points, and more if errors cluster.
Why do long agent trajectories fail more?
"Reliability compounds multiplicatively: 95 percent per step is 60 percent over ten steps and 8 percent over fifty. So I shorten the path, verify and retry steps, and checkpoint. The numbers assume independent steps, which overstates the effect when steps are correlated and understates it when one early mistake poisons the rest." Follow-up: "What does a retry cost?" Extra calls and latency, and it only helps if the failure is detected.
What is the difference between pass@k and pass^k?
A new prompt scores 3 points higher. Do you ship it?
"Not yet. I check the suite size, the interval, and whether the gain exceeds the run-to-run noise measured by rerunning the old prompt. With a small suite the same agent can show that gap by chance. I use paired comparison on the same cases and more trials." Follow-up: "What if you cannot afford more runs?" Prefer a rollout with monitoring (module 9) over a decision on weak evidence.
Common mistakes
One run per case reported as the score.
No interval, and no suite size.
Comparing two agents on different cases or seeds.
Treating pass@k as reliability.
Assuming trials are independent when failures cluster by task.
Unfrozen environments: live web pages change between runs.
Reading differences below the noise floor as improvements.
Tuning prompts on the test set, until the score only measures the tuning (module 7 and 10).
What the example does not do
No real agent or model; the success draws are Bernoulli.
Independent tries given the task; real tries share causes (a bad tool, a confusing instruction).
A fixed suite in each experiment; the variation from choosing different tasks is only partly captured.
A perfect retry detector in the chain table.
Optional real experiment (I did not run this): run one agent 10 times on 30 tasks, then compute the suite score for each run, the interval, pass@5 and pass^5, and the share of tasks that are inconsistent.
Check yourself
A task passes 80 percent of the time per try. Your agent gets four tries. What is pass@4, and what is the chance all four tries pass? (pass@4 is 0.998; all four pass is 0.41. The first says it can ever solve it; the second says you can rely on it.)
For the curious Why a small suite crowns false winners
Two identical agents, one run each: on 20 tasks one looked at least 3 points better 84 percent of the time; on 300 tasks, 35 percent. The standard deviation of one suite score was 0.077 at 20 tasks and 0.021 at 300. Before you believe a gap, run the same version twice and keep that difference as your noise floor. These numbers come from a simulation with assumed task difficulty.
What you will use daily at work
Practice
What it looks like
Report a score with its interval
80 percent on 10 cases is 49 to 94 percent; on 100 cases it is 71 to 87.
Run each case 3 to 5 times
Repeats cut the noise and let you see unreliable cases.
Keep a noise baseline
Run the same version twice; changes below that gap are noise.
Say how big the suite was
Put the case count next to every score.
Draft: confirm how your eval tool records repeated runs and what seeds your provider allows.
What you must be able to say (one page)
An agent score is a measurement with an error bar. Four things vary: the model's samples, the task set you happened to pick, the environment (web pages, APIs, time), and the grader. Agent trajectories are non-deterministic and heterogeneous: the same input can lead to significantly different trajectories.
Report pass@k and pass^k, not one run.pass@k: a problem counts as solved if any of k samples passes. For a task with a 0.8 chance per try: pass@4 is 0.998 but all four tries pass (called pass^k, the notation used by tau-bench) only 0.41. Use pass@k for "can it ever", and pass^k for "can I rely on it".
Long trajectories compound. If each step works with probability 0.95, ten steps all work with 0.60, twenty with 0.36, fifty with 0.08. At 0.99 per step, fifty steps give 0.61. One automatic retry (with perfect detection) lifts 0.95-per-step, 20 steps from 0.36 to 0.95; real detection is imperfect, so this is an upper bound.
Small suites produce false winners. Two identical agents, one run each: on 20 tasks one looked at least 3 points better 84 percent of the time; 50 tasks: 70; 100: 62; 300: 35. The standard deviation of one agent's suite score was 0.077 (20 tasks), 0.052 (50), 0.037 (100), 0.021 (300).
Intervals tell you how much to believe a number. A measured 80 percent is 49 to 94 percent on 10 cases, 71 to 87 on 100, 77 to 82 on 1,000 (95 percent Wilson interval). To get within 5 points you need about 246 independent trials.
Repeating trials per task helps detection: a true 5-point drop was seen as lower in 70 percent of comparisons with 50 tasks and one run each, 93 percent with five runs each, and 99.9 percent with 200 tasks and five runs.
Honest limits: every number is from a simulation with assumed task difficulty, and the "independent trials" assumption is wrong for real agents whose errors cluster by task.