CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 3 of 12 · 2 min read

The people who bother to give feedback are not a random sample of your users

If unhappy users speak up more than happy ones, the feedback you collect describes the complainers, not the whole population.

Think of a restaurant comment card box. The people who write in are disproportionately the ones who had a terrible meal or an amazing one. They are not the much larger group who thought it was fine and said nothing.

In plain words: response bias means the people who give feedback differ systematically from those who don't, and inverse-propensity weighting corrects for this by weighting each response by one over its probability of being observed -- but that correction only works if you actually know, or can estimate, those observation probabilities, which is usually the hard part in practice.

A worked example. In a simulated population with 80% true satisfaction, unhappy users give feedback at a 6% rate while happy users give feedback at only 1.5% -- so despite satisfaction being high, the raw thumbs-up share among the people who actually responded is just 50%, a 30-point understatement. Applying inverse-propensity weighting, which multiplies each response by the inverse of its own group's known feedback rate, recovers the true 80% almost exactly -- but only because the simulation handed the correction the exact feedback rates it needed, which a real team usually has to estimate from something like an independent survey.

The key lines. From code/mini-feedback/feedback.py:

# from code/mini-feedback/feedback.py
# inverse-propensity correction, valid only if the two feedback rates are KNOWN
w_up, w_down = 1 / p_fb_happy, 1 / p_fb_unhappy
rew.append(up * w_up / (up * w_up + down * w_down))

Read it as: the comment above the code states its own precondition in capital letters. "Valid only if the two feedback rates are KNOWN" is the module being honest about where this fix becomes hard in the real world. The arithmetic of inverse-propensity weighting is one line. But the feedback rates it needs are the actual research problem.

Response bias, inverse-propensity weighting

Response bias is that those who give feedback differ from those who do not. Inverse-propensity weighting weights each response by one over its probability of being observed; it needs those probabilities.

Example: a raw 50% thumbs-up share sitting 30 points below the true 80% satisfaction is response bias made visible -- the fix isn't "read the number more carefully," it's "know, or estimate, how differently happy and unhappy users behave" before trusting any raw feedback share.