Part 10 of 12 · 1 min read
Quick notes
- Feedback is evidence about quality, but a biased and sparse sample. Make it cheap and specific: a reason code with the thumb, an edit box.
- Response bias means the raw thumbs-up share is wrong when unhappy users answer more often, and re-weighting only works if you know the response rates.
- Signals are noisy detectors: a downvote finds few of the real failures, regeneration finds more at lower precision. Use them to find candidates and trends, not as ground truth.
- Winner's curse: picking the best of many prompts by a few hundred votes overstates the gain, and optimising a proxy such as agreement can lower correctness; expert evaluators and launch-blocking criteria are the remedy.
- A good loop: collect signals, triage and join to the trace, let experts label a sample, fix, add the case to the eval set, verify through the CI gate, roll out gradually, then watch.