Part 10 of 12 · 1 min read
Quick notes
- Preference tuning pushes a model toward outputs people prefer, using comparisons rather than gold answers. RLHF fits a reward model and then runs RL. DPO trains on the pairs directly.
- The reward-model loss is Bradley-Terry: minimise
-log sigmoid(r(winner) - r(loser)). Only differences matter. - The objective is reward minus beta times the KL from the reference model, and its maximiser has a closed form. DPO comes from that closed form, which is why it needs no reward model.
- beta is a leash: small beta gains more reward but trusts the reward model more. In the toy RLHF and DPO matched once the reward model's scale was fitted fairly. A method comparison can be an artefact of such a setting.
- Noisy labels hurt both, and a learned reward can be over-optimised (Goodhart). Best-of-N is the cheap alternative. Evaluate on held-out human or verifier judgements, not on the reward model.