CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 10 of 12 · 1 min read

Quick notes

  • Preference tuning pushes a model toward outputs people prefer, using comparisons rather than gold answers. RLHF fits a reward model and then runs RL. DPO trains on the pairs directly.
  • The reward-model loss is Bradley-Terry: minimise -log sigmoid(r(winner) - r(loser)). Only differences matter.
  • The objective is reward minus beta times the KL from the reference model, and its maximiser has a closed form. DPO comes from that closed form, which is why it needs no reward model.
  • beta is a leash: small beta gains more reward but trusts the reward model more. In the toy RLHF and DPO matched once the reward model's scale was fitted fairly. A method comparison can be an artefact of such a setting.
  • Noisy labels hurt both, and a learned reward can be over-optimised (Goodhart). Best-of-N is the cheap alternative. Evaluate on held-out human or verifier judgements, not on the reward model.