CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • RLVR gets its reward from a program (a test, a checker, a validator): sample attempts, grade them, raise the probability of the ones that passed. The model optimises the checker, not your intent.
  • GRPO uses a group-relative advantage ((reward minus group mean) over group standard deviation), so no value model is needed. At the same learning rate it was faster than REINFORCE in the toy; at each method's best rate they nearly tied, so tune the rate before crediting the algorithm.
  • No signal, no learning: a group where every attempt gets the same reward has zero advantage. Use a curriculum, larger groups or partial credit, and remember that partial credit is another thing to hack.
  • Reward hacking is the central risk. A weak verifier or a loophole sends the verifier pass rate to near 100 percent while true success collapses, and a KL penalty slows honest learning before it stops the hack.
  • For agents, RL needs an environment you can reset and score cheaply and a verifier you trust, plus an independent held-out check tracked next to the training reward.