CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 6 of 12 · 1 min read

What the example found

Feedback rate and bias, flag precision/recall, winner's curse at two vote counts, and the proxy-optimization trade-off.

How to read the bias-and-flags table: each row is a different simulated measurement; the first two rows show the raw feedback population, the next two show two implicit signals scored as failure detectors, and the last two show winner's curse at two different vote counts.

Experiment Result
Feedback rate 2.4 percent of users
True satisfaction / naive thumbs-up share / re-weighted 80% / 50% / 80%
Downvote as failure flag precision 72%, recall 5%
Regeneration as failure flag precision 53%, recall 36%
Best of 10 by 100 votes: observed / true 78.1% / 72.6%
Best of 10 by 1,000 votes: observed / true 75.0% / 74.2%

What to notice: only 2.4% of users leave any feedback at all. That tiny, self-selected sliver is biased enough to misreport satisfaction by 30 points before any correction. It is the single number that makes the loudest case for not reading raw thumbs as a satisfaction score.

How to read the agreeableness table: each row is a different assumed weight users place on agreeableness (versus correctness) in their thumbs; the columns compare the correctness of the variant selected by thumbs against the variant selected by an expert.

Weight users put on agreeableness Correctness, chosen by thumbs Chosen by expert
0.0 84% 84%
0.3 66% 84%
0.5 61% 84%
0.7 59% 84%

What to notice: the expert-chosen column never moves at all across the table, which is the real point. The thumbs-chosen column's decline from 84% to 59% is not the agent getting worse at the task. It is the selection process itself drifting toward whatever users find agreeable, entirely independent of correctness.

{{svg:loop|Five stages: collect, triage, fix, verify, watch.}} {{svg:results|Response bias, proxy optimisation, flag quality and winner's curse.}}