Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-bench.
Done when: you report the top-1 rate at 100 and 1,000 tasks.
Easy 2: Read a leaderboard (30 minutes)
Pick one public leaderboard row for an agent and note its harness, number of attempts, cost and date.
Done when: you list three things that would change before you could compare it with your agent.
Easy 3: Ceiling (30 minutes)
If 15 percent of 500 tasks are unpassable, what is the ceiling and what is the largest observable gap between two agents with true skill 0.9 and 0.8?
Done when: you compute both.
Hint
ceiling is the passable share; the gap is at most that share times 0.1.
Medium
Medium 1: Power curve (2 hours)
Plot power against n for a 3-point gap.
Done when: you give the n that reaches 80 percent power.
Hint
reuse power(a, b, n).
Medium 2: Contamination detector (3 hours)
Add a "verbatim recall" probe to the simulation and measure its true and false positive rates.
Done when: you report them and the effect on ranking when flagged items are removed.
Hint
a verbatim probe asks the model to complete the first half of a task.
Medium 3: Repeated trials (2 hours)
Run each agent three times per task and average.
Done when: you show the change in power against one run per task.
Hint
average per-task results over runs before the bootstrap.
Hard
Hard 1: Reproduce a row (8 hours)
Re-run a public agent on a small slice of a public benchmark with your own harness. (I did not run this, so your numbers are the ground truth.)
Done when: you report the difference from the published number and list the causes.
Hard 2: Private split (5 hours)
Build a 50-task private benchmark from your own traces and compare model ranking with a public one. (I did not run this.)
Done when: you report the rank correlation.
Hard 3: Audit (6 hours)
Read 30 tasks from a public benchmark and count flawed tests or ambiguous descriptions. (I did not run this.)
Done when: you report the flaw rate with an interval and compare it with the 59.4 percent OpenAI reported for its hard subset.
Hint
keep a written rubric for "flawed" and have a second person recheck 10 tasks.