Part 10 of 12 · 1 min read
Quick notes
- RLVR gets its reward from a program (a test, a checker, a validator): sample attempts, grade them, raise the probability of the ones that passed. The model optimises the checker, not your intent.
- GRPO uses a group-relative advantage ((reward minus group mean) over group standard deviation), so no value model is needed. At the same learning rate it was faster than REINFORCE in the toy; at each method's best rate they nearly tied, so tune the rate before crediting the algorithm.
- No signal, no learning: a group where every attempt gets the same reward has zero advantage. Use a curriculum, larger groups or partial credit, and remember that partial credit is another thing to hack.
- Reward hacking is the central risk. A weak verifier or a loophole sends the verifier pass rate to near 100 percent while true success collapses, and a KL penalty slows honest learning before it stops the hack.
- For agents, RL needs an environment you can reset and score cheaply and a verifier you trust, plus an independent held-out check tracked next to the training reward.