Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run pytest -q and python run_all.py in code/mini-flaky.
Done when: you report the false-winner rate at 20 and 300 tasks and explain it.
Easy 2: Interval by hand (30 minutes)
Compute the 95 percent Wilson interval for 36 passes out of 45.
Done when: it matches wilson(36, 45).
Hint
wilson(k, n) returns the lower and upper limit.
Easy 3: Compound a trajectory (30 minutes)
A 12-step agent has 97 percent per-step reliability.
Done when: you compute the end-to-end success and the per-step reliability needed for 90 percent.
Hint
0.97 to the power 12; for 90 percent solve r = 0.9 to the power 1/12.
Medium
Medium 1: Cluster the errors (2 hours)
Add a shared per-task failure cause so that tries are correlated.
Done when: you show how the interval for the suite score widens against independent tries.
Hint
draw a per-task failure that applies to every try of that task.
Medium 2: Noise floor (2 hours)
Compute the run-to-run difference distribution for one agent and set a "significant change" threshold at the 95th percentile.
Done when: you report the threshold for 30, 100 and 300 tasks.
Hint
run the suite twice many times and take percentiles of the absolute difference.
Medium 3: Allocate trials (3 hours)
With a budget of 500 runs, compare 500 tasks x 1 run with 100 tasks x 5 runs for detecting a 5-point drop.
Done when: you report detection rates and say which allocation you would pick.
Hint
reuse detect_drop(n, k).
Hard
Hard 1: Measure your own agent (6 hours)
Run a real agent 10 times on 30 tasks. (I did not run this, so your numbers are the ground truth.)
Done when: you report suite-score spread, pass@5, pass^5 and the share of inconsistent tasks, with the model and date.
Hard 2: Environment noise (5 hours)
Compare runs with a live tool and with recorded tool outputs. (I did not run this.)
Done when: you report the variance removed by recording.
Hint
log tool inputs and outputs and replay them.
Hard 3: Paired against unpaired (4 hours)
Implement a paired bootstrap for two versions on the same cases. (I did not run this.)
Done when: you show the interval is narrower than an unpaired comparison on the same data.
Hint
resample cases, not individual runs.