Part 6 of 12 · 2 min read
What the example found
pass@k vs pass^k, compounding over steps, interval width by sample size, and detection rates for a real drop.
How to read the consistency table: each row is a different number of tries k, for a task with an 80% chance of success per try.
| k | pass@k | pass^k |
|---|---|---|
| 1 | 0.80 | 0.80 |
| 2 | 0.96 | 0.64 |
| 4 | 0.998 | 0.41 |
| 8 | 1.000 | 0.17 |
What to notice: as k rises, pass@k climbs toward certainty while pass^k falls toward zero. The same task and the same agent produce two numbers heading in opposite directions. This is the clearest illustration of why reporting only one of them misleads.
How to read the compounding table: each row is a trajectory length in steps; the columns are the end-to-end success chance at different per-step reliabilities.
| Steps | 95% per step | 99% per step | 95% with one retry |
|---|---|---|---|
| 1 | 0.95 | 0.99 | 0.997 |
| 5 | 0.77 | 0.95 | 0.988 |
| 10 | 0.60 | 0.90 | 0.975 |
| 20 | 0.36 | 0.82 | 0.951 |
| 50 | 0.08 | 0.61 | 0.882 |
What to notice: at 50 steps, a per-step reliability that sounds excellent (95%) compounds down to just 8% end-to-end success. A single retry with perfect detection pulls that back up to 88%. That is why shortening trajectories and adding verified retries matters more than squeezing out a slightly better model.
How to read the interval table: each entry is the 95% Wilson interval around a measured 80% success rate, at a different sample size.
| n | Interval for a measured 80% |
|---|---|
| 10 | 0.49 to 0.94 |
| 30 | 0.63 to 0.90 |
| 100 | 0.71 to 0.87 |
| 300 | 0.75 to 0.84 |
| 1000 | 0.77 to 0.82 |
What to notice: the interval at n=10 is wide enough to be nearly useless (anywhere from under half to almost all). Even n=1000 still leaves a real range. There is no sample size at which the interval collapses to a single point. There is only one at which it becomes narrow enough to act on.
How to read the detection table: each row is a combination of task count and repeats per task; the value is how often a true 5-point quality drop was correctly detected as lower.
| Setup | Detection rate |
|---|---|
| 50 tasks, one run | 0.70 |
| 50 tasks, five runs | 0.93 |
| 200 tasks, one run | 0.90 |
| 200 tasks, five runs | 0.999 |
What to notice: going from one run to five runs on the same 50 tasks (0.70 to 0.93) buys nearly as much detection power as quadrupling the task count at one run each (0.70 to 0.90). Repeats and tasks are both valid ways to shrink the noise. The module's own rule is that you usually need some of both.
{{svg:noise|The model, the task set and the environment each add noise; the grader adds a fourth.}} {{svg:results|Two identical agents look different on small suites; per-step reliability compounds over long trajectories.}}