Part 8 of 12 · 2 min read
Answering preference-tuning questions in an interview
Explain RLHF and DPO, and how they differ.
"RLHF trains a reward model on comparisons with a Bradley-Terry loss, then optimises the policy, usually with PPO, to maximise reward minus beta times KL from the reference. DPO starts from the closed-form optimum of that same objective, pi* proportional to pi_ref times exp(reward over beta). It trains the policy directly on the pairs with a logistic loss on the log-ratio difference, so it needs no reward model and no sampling loop. In the example's toy, with a well-fitted reward model, both reached the same true reward." Follow-up: "When would you still use RLHF?" When you want a reusable reward model (ranking, filtering, best-of-N) or online data from the current policy.
What does beta do?
"It weights the KL penalty. Large beta keeps the policy near the reference. Small beta lets it chase the reward and its errors. In the example's toy beta 0.05 gave 2.14 true reward with a KL of 1.5 and beta 3 gave 1.18 with a KL of 0.06." Follow-up: "How do you choose it?" On a held-out evaluation that is independent of the reward model, watching for the point where gains stop.
What is reward hacking or over-optimisation?
"Optimising a proxy until it diverges from the real goal. In the example's toy, with 100 noisy pairs and a small beta, the learned reward rose from 0.2 to 3.2 while the true reward fell below zero in the worst run. 60 percent of runs ended worse than the starting model." Follow-up: "How do you mitigate it?" Larger KL penalty, more and cleaner data, reward-model ensembles or uncertainty, early stopping on a held-out check, and mixing in verifiable rewards.
Why train on comparisons instead of scores?
"People disagree on absolute scores and drift over time, but they agree more often on which of two responses is better. The Bradley-Terry loss turns rankings into a scoring model. About 73 percent agreement was reported among InstructGPT labelers, so labels are still noisy, which my toy shows costs a lot of reward at 30 percent flips." Follow-up: "What does noise do?" It lowers the achievable reward and can make the reward model confidently wrong.