CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that mask over-optimisation

Common mistakes, and what the example does not do.

Mistakes that mask over-optimisation

  • Comparing RLHF and DPO without matching reward-model calibration (the first run did this).
  • Evaluating on the reward model that the policy was trained against.
  • Tiny beta with a weak reward model.
  • Skipping the reference model or letting it drift.
  • Training on contradictory or noisy preference pairs without auditing.
  • Ignoring regressions in safety, tool use and factuality.
  • Treating the implicit DPO reward as an absolute score (it is defined up to a per-prompt constant).
  • Assuming PPO behaves like exact gradients; real RL adds variance and instability.
  • Using best-of-N with a poorly calibrated reward model and large N.
  • Assuming preferences are universal.

What the example does not do

About this example What the example does not do
  • No language model: a lookup-table policy over 10 responses per prompt, with no generalisation between prompts.
  • No PPO and no sampling noise, so none of RL's instability or variance appears.
  • True rewards are random, and the labelers are perfect Bradley-Terry samplers (plus optional flipped labels). Real labels have bias, length effects and annotator disagreement.
  • The reward model is also tabular, so it cannot extrapolate to unseen prompts or responses, which is where real over-optimisation often starts.
  • Ten random worlds per setting, a single reference policy family, and hand-chosen learning rates.
  • No offline-versus-online distinction, no length bias, no RLAIF.
  • Optional real experiment: with a small open model and a public preference dataset, train DPO at beta 0.05, 0.1 and 0.5 using a library such as TRL. Then compare win rate against the SFT model by a held-out judge, and check for length and refusal drift.

Check yourself

What do beta in RLHF and beta in DPO control?

Show answer

How far the tuned policy may move from the reference model. Small beta lets it gain more reward and drift further. Large beta keeps it close. In the toy the perfect policy at beta 0.05 had KL 1.52 from the reference, at beta 3 only 0.06.

For the curious A learned reward is a proxy, and it can be gamed

With 100 pairs, 30 percent label noise and beta 0.02, 60 percent of 20 runs ended worse than the reference on the true reward. In the worst run the learned reward rose from 0.20 to 3.24 while the true reward fell from 0.85 to -0.39. The first run also showed DPO beating RLHF, and the cause was the reward model's L2 setting, not the method. The policy is a table, there is no PPO and no language model.

What you will use daily at work

Practice What it looks like
Collect comparisons, not scores And measure labeler agreement.
Hold out a check the reward model never saw Human review or a verifier.
Use best-of-N first It needs no training, only N times the inference.
Run a regression eval before and after Safety, tool use, hallucination, format.