CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 7 of 12 · 1 min read

How to build it so it holds up

  • Start from a strong SFT model and keep it as the reference.
  • Collect comparisons, not scores, and measure labeler agreement (a figure of about 73 percent was reported for InstructGPT).
  • Pick beta by a held-out evaluation, not by the training reward.
  • Watch for over-optimisation: track the proxy and an independent check together, and stop early.
  • Check reward-model calibration (scale and regularisation) before concluding that one method is better.
  • Filter noisy or contradictory pairs; audit a sample by hand.
  • Prefer DPO for simplicity; use RLHF or best-of-N when you need a reusable scorer.
  • Do not trust the reward model on prompts unlike its training data.
  • Run a regression eval (safety, tool use, hallucination) before and after. Preference tuning can improve overall preference while making one of these worse.
  • Version datasets, reward models and reference models together.