Part 2 of 12 · 2 min read
A reward model learns from which answer won, not how good either was
People disagree on absolute scores but agree more often on which of two answers is better, so train on the comparison instead of the score.
Think of judging a baking contest: two judges might give wildly different absolute scores to the same cake, but they'll usually agree on which of two cakes placed in front of them is better.
In plain words: a reward model doesn't try to learn an absolute quality score. It's trained so that, for each pair a person compared, the winner's score comes out higher than the loser's, using exactly the Bradley-Terry formula for the probability that one beats the other.
A worked example. Even using the true reward (not a learned approximation) to generate Bradley-Terry labels, the resulting labels only agree with "the actually-higher-reward response won" about 77 percent of the time. Bradley-Terry labeling is inherently probabilistic, not a clean ranking. A reward model fit from only 300 real comparison pairs reaches 71 percent pairwise accuracy, close to that ceiling given how little data it saw.
The key lines. From code/mini-preference/pref.py:
# from code/mini-preference/pref.py
def train_reward_model(data, steps=3000, lr=1.0, l2=1e-4):
"""Tabular Bradley-Terry reward model: minimise -mean log sigmoid(r[c,w] - r[c,l]) + l2*||r||^2."""
c, w, l = data
if len(c) == 0:
raise ValueError("no comparison pairs to fit")
R = np.zeros((C, A)); n = len(c)
for _ in range(steps):
z = R[c, w] - R[c, l]
g = -(1 - sigmoid(z)) / n
Read it as: the gradient g only ever looks at the difference R[c,w] - R[c,l] between a pair's two scores. The loss never asks the model to predict an absolute number, only to get the relative order right, which is the entire Bradley-Terry idea in two lines.
Preference data, reward model, Bradley-Terry model
Preference data is (prompt, winning response, losing response) triples from humans, AI judges or verifiers. A reward model maps (prompt, response) to a scalar score, trained so winners score higher than losers. The Bradley-Terry model says P(a beats b) equals sigmoid(reward(a) minus reward(b)). The basis of the reward-model loss above.
Example: the 77 percent accuracy ceiling above is a direct consequence of Bradley-Terry's own probabilistic model. Even a perfect reward model can't make noisy human-style comparisons agree with the ranking 100 percent of the time.