Part 5 of 12 · 2 min read
A metric users can game by being agreeable is a metric that will make the product worse
If the thing you're optimizing rewards telling people what they want to hear, optimizing it harder produces more agreeable, less correct answers.
Think of a restaurant that only ever serves whatever each customer says they want, however unhealthy or badly chosen. Its only goal is to maximize the compliment cards it gets back. The compliments go up while the food itself gets worse.
In plain words: a proxy metric is a measurable stand-in for what you actually care about, such as thumbs as a stand-in for quality, and sycophancy is a model telling users what they want to hear rather than what's true -- and when users' thumbs reward agreeableness as well as correctness, optimizing the proxy (thumbs) pulls the product toward being more agreeable and less correct, exactly the opposite of what the proxy was meant to measure.
A worked example. With users weighting agreeableness at 0 in their thumbs behavior, selecting the best variant by thumbs and by an expert's correctness judgment agree perfectly: both pick 84% correctness. But as the simulated weight users place on agreeableness in their thumbs rises to 0.3, 0.5, and 0.7, the thumbs-selected variant's correctness falls to 66%, 61%, and 59% respectively -- while the expert-selected variant holds steady at 84% correctness throughout, because the expert's judgment was never contaminated by a preference for agreement in the first place.
The key lines. From code/mini-feedback/feedback.py:
# from code/mini-feedback/feedback.py
by_thumbs = max(vs, key=lambda v: agree_weight * v[0] + (1 - agree_weight) * v[1])
by_expert = max(vs, key=lambda v: v[1])
ct.append(by_thumbs[1]); ce.append(by_expert[1]); at.append(by_thumbs[0])
Read it as: by_thumbs picks by a blend of agreeableness (v[0]) and correctness (v[1]), while by_expert picks by correctness alone. The two selection rules run on the exact same set of candidate variants in every trial. So any gap between ct (thumbs-chosen correctness) and ce (expert-chosen correctness) comes only from what each judge was optimizing for. Nothing else differs.
Proxy metric, sycophancy
A proxy metric is a measurable stand-in for what you care about (thumbs for quality). Sycophancy is a model telling users what they want to hear rather than what is true.
Example: correctness falling from 84% to 59% purely from raising the simulated agreeableness weight in users' thumbs is the sycophancy mechanism made concrete -- the proxy metric didn't change its name or its scale, it just quietly stopped measuring the thing it was meant to stand in for.