CSE Roadmap

Specialist tier · about 5 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that let a policy game the verifier

Common mistakes, and what the example does not do.

Mistakes that let a policy game the verifier

  • Trusting the verifier without testing for shortcuts.
  • Watching only the training reward.
  • Comparing algorithms at one learning rate.
  • Training on prompts that are always passed or always failed.
  • Groups too small for the pass rate.
  • Leaving the environment writable (tests, scoring files, network).
  • Using a KL penalty as a substitute for a better verifier.
  • Adding process rewards without checking they correlate with real success.
  • Skipping SFT so the starting policy never produces a success.
  • Treating a toy result as evidence about frontier models.

What the example does not do

About this example What the example does not do
  • No language model: independent softmaxes per step, no text, no context-dependence, one correct chain per prompt.
  • No value network, no per-token ratios, and clipping is applied at the step level on a tabular policy.
  • The weak verifier and loophole are chosen for the example; the loophole is found by chance in about a quarter of early samples, which makes hacking fast.
  • Learning rates were chosen from a coarse grid; results at the best rate differ by small amounts that were not tested for significance.
  • The explanation for the batch-baseline failures is a guess.
  • No exploration problem beyond sparse reward, no multi-turn state, no tool noise, no cost model.
  • Optional real experiment: train a small open model with a library such as TRL or a similar GRPO implementation on a task with unit-test rewards (for example short Python functions), track the held-out pass rate and the share of groups with signal, and add a deliberately weak test to see whether the model learns to game it.

Check yourself

A group of 8 attempts at one prompt all fail. How much does it teach the policy?

Show answer

Nothing: every advantage is zero, and the test checks the gradient is exactly zero. At a 1.3 percent pass rate only 9.6 percent of groups of 8 have both a pass and a fail.

For the curious The policy optimises the checker, not your intent

A weak verifier that checked only the last step reached verifier pass 0.999 but true success 0.075. A SKIP loophole took verifier pass from 0.35 to 0.99 in 10 updates while true success fell to 0.000. A KL penalty slowed honest learning before it slowed the hack. At the same learning rate GRPO was 2 to 3 times faster than REINFORCE, but at each method's best rate they nearly tied, so tune the rate per method first. This is a tabular toy, not a language model.

What you will use daily at work

Practice What it looks like
Build the environment and verifier first Resettable, cheap to score, hard to game.
Hold out an independent check A different verifier or a human audit.
Track true success next to training reward A gap between them is a warning.
Try SFT, preference tuning and best-of-N before RL RL is data-hungry for long tasks.