Part 9 of 12 · 2 min read
Mistakes that let a policy game the verifier
Common mistakes, and what the example does not do.
Mistakes that let a policy game the verifier
- Trusting the verifier without testing for shortcuts.
- Watching only the training reward.
- Comparing algorithms at one learning rate.
- Training on prompts that are always passed or always failed.
- Groups too small for the pass rate.
- Leaving the environment writable (tests, scoring files, network).
- Using a KL penalty as a substitute for a better verifier.
- Adding process rewards without checking they correlate with real success.
- Skipping SFT so the starting policy never produces a success.
- Treating a toy result as evidence about frontier models.
What the example does not do
About this example What the example does not do
- No language model: independent softmaxes per step, no text, no context-dependence, one correct chain per prompt.
- No value network, no per-token ratios, and clipping is applied at the step level on a tabular policy.
- The weak verifier and loophole are chosen for the example; the loophole is found by chance in about a quarter of early samples, which makes hacking fast.
- Learning rates were chosen from a coarse grid; results at the best rate differ by small amounts that were not tested for significance.
- The explanation for the batch-baseline failures is a guess.
- No exploration problem beyond sparse reward, no multi-turn state, no tool noise, no cost model.
- Optional real experiment: train a small open model with a library such as TRL or a similar GRPO implementation on a task with unit-test rewards (for example short Python functions), track the held-out pass rate and the share of groups with signal, and add a deliberately weak test to see whether the model learns to game it.
Check yourself
A group of 8 attempts at one prompt all fail. How much does it teach the policy?
Show answer
Nothing: every advantage is zero, and the test checks the gradient is exactly zero. At a 1.3 percent pass rate only 9.6 percent of groups of 8 have both a pass and a fail.
For the curious The policy optimises the checker, not your intent
A weak verifier that checked only the last step reached verifier pass 0.999 but true success 0.075. A SKIP loophole took verifier pass from 0.35 to 0.99 in 10 updates while true success fell to 0.000. A KL penalty slowed honest learning before it slowed the hack. At the same learning rate GRPO was 2 to 3 times faster than REINFORCE, but at each method's best rate they nearly tied, so tune the rate per method first. This is a tabular toy, not a language model.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Build the environment and verifier first | Resettable, cheap to score, hard to game. |
| Hold out an independent check | A different verifier or a human audit. |
| Track true success next to training reward | A gap between them is a warning. |
| Try SFT, preference tuning and best-of-N before RL | RL is data-hungry for long tasks. |