Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-rlvr (the second takes about two minutes).
Done when: you can read each table and say which claim it supports.
Easy 2: Signal at the start (30 minutes)
For a pass rate of 2 percent compute the chance a group of 8 has both outcomes, then check by simulation.
Done when: your numbers agree with group_has_signal.
Hint
the formula is 1 - p^G - (1-p)^G.
Easy 3: Find the shortcut (30 minutes)
Write a verifier for a task of your own and list three ways a policy could pass it without solving the task.
Done when: you have changed the verifier to close at least one.
Medium
Medium 1: Curriculum (3 hours)
Add prompts of different difficulty and train on those with pass rates between 10 and 90 percent.
Done when: you show faster learning than uniform sampling.
Hint
estimate the pass rate per prompt from a first batch of rollouts.
Medium 2: Learning rate fairness (3 hours)
Repeat the method comparison on a finer grid and 50 seeds with confidence intervals.
Done when: you say which differences are larger than seed noise.
Medium 3: Audit check (3 hours)
Add a second, independent verifier that is evaluated but never trained against, and plot it beside the training reward.
Done when: it reveals the loophole before the true success hits zero.
Hint
keep the audit verifier's code and data away from the training loop.
Hard
Hard 1: GRPO on a small model (8 hours)
Train a small open model on a code or math task with a verifiable reward and report held-out pass rate and share of groups with signal. (I did not run this, so your numbers are the ground truth.)
Done when: you show the learning curve and the group-signal curve.
Hint
start with a model that already passes a few percent of your prompts.
Hard 2: Break the verifier (8 hours)
Give the model a task whose tests it can modify and measure how quickly hacking appears. (I did not run this.)
Done when: you show the failure and a fix that closes it.
Hard 3: Agent environment (8 hours)
Build a small tool-use environment with a resettable state and a pass/fail check and train a policy on it. (I did not run this.)
Done when: you report cost per rollout and the pass rate against rollouts.