Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-papers.
Done when: you can explain each table in the results.
Easy 2: Fill the template (45 minutes)
Choose a paper you know and write its claim, setup, evidence and catch in a PaperNote.
Done when: red_flags runs and you can say which flags you accept and which you reject.
Easy 3: Bootstrap your own set (30 minutes)
Take your own evaluation results and compute a 95 percent bootstrap interval for accuracy.
Done when: you report the width and whether your last improvement exceeded it.
Medium
Medium 1: Correlated benchmarks (3 hours)
Make the m benchmarks share difficulty and recompute the chance of a significant win on at least one.
Done when: you plot the rate against m for correlation 0, 0.5 and 1.
Medium 2: Tuning budget in your task (3 hours)
Measure the score spread across 20 prompt or setting variants on your task and compute the gap a 20-try method would show against a single-try baseline.
Done when: you compare the measured gap with the simulation.
Medium 3: Choose items to annotate (3 hours)
Given a budget of 500 annotations, decide between one large random set and several slices, and justify with interval widths.
Done when: you report the interval for each slice and overall.
Hard
Hard 1: Reproduce a headline (8 hours)
Pick a published method that can be tried on a small model or dataset, reproduce its headline with equal tuning budgets and five seeds, and write up claim, setup, evidence and catch. (I did not run this, so your numbers are the ground truth.)
Done when: you report the gap with an interval and whether it matches the paper.
Hard 2: Audit three papers (8 hours)
Run the checklist on three recent agent papers and write which flags were genuine after reading the appendix. (I did not run this.)
Done when: you list the flags confirmed, dismissed and unresolved with page references.
Hard 3: Significance workbench in PyTorch (8 hours)
Re-implement the equal-model simulation with real fine-tuning runs of a small network under different seeds and compare the spread with my assumed values. (I did not run this.)
Done when: you report the measured seed spread and the false-gain rate for 1, 3 and 5 runs.