Part 7 of 12 · 1 min read
How to build it so it holds up
- Start from a strong SFT model and keep it as the reference.
- Collect comparisons, not scores, and measure labeler agreement (a figure of about 73 percent was reported for InstructGPT).
- Pick beta by a held-out evaluation, not by the training reward.
- Watch for over-optimisation: track the proxy and an independent check together, and stop early.
- Check reward-model calibration (scale and regularisation) before concluding that one method is better.
- Filter noisy or contradictory pairs; audit a sample by hand.
- Prefer DPO for simplicity; use RLHF or best-of-N when you need a reusable scorer.
- Do not trust the reward model on prompts unlike its training data.
- Run a regression eval (safety, tool use, hallucination) before and after. Preference tuning can improve overall preference while making one of these worse.
- Version datasets, reward models and reference models together.