Part 2 of 12 · 2 min read
An implicit signal is plentiful but noisy; measure its precision and recall before trusting it
A downvote or a regeneration is a hint, not a verdict -- and the only way to know how good a hint it is, is to check it against real failures.
Think of a smoke detector that sometimes goes off from cooking toast and sometimes stays silent during an actual small fire. It is useful, but only once you know how often each kind of mistake happens.
In plain words: explicit feedback is users answering a direct request (thumbs, stars, a form), implicit feedback is inferred from behaviour (regeneration, rephrasing, abandonment) -- and treating either kind as ground truth without first measuring its precision (of the things it flags, how many are real failures) and recall (of the real failures, how many it flags) means you're trusting a detector whose actual accuracy you never checked.
A worked example. Using a downvote as a failure flag, against a simulated population where 20% of responses genuinely fail: precision is 72% (most downvotes really do mark failures) but recall is only 5% (it catches almost none of the real failures that happened). Using regeneration as the flag instead gives precision 53% and recall 36% -- lower precision, but more than seven times the recall. Neither signal alone gives you the real failure rate; each is a different, specific trade-off between catching real problems and raising false alarms.
The key lines. From code/mini-feedback/feedback.py:
# from code/mini-feedback/feedback.py
def downvote_as_failure_flag(n=50000, fail_rate=0.20, p_down_fail=0.05, p_down_ok=0.004, seed=4):
"""Precision and recall of 'user downvoted' as a detector of real failures, plus share of failures never flagged."""
rng = random.Random(seed)
tp = fp = fn = 0
for _ in range(n):
fail = rng.random() < fail_rate
down = rng.random() < (p_down_fail if fail else p_down_ok)
tp += fail and down; fp += (not fail) and down; fn += fail and not down
return tp / (tp + fp), tp / (tp + fn), tp + fp
Read it as: the function treats a downvote as a classifier output. It scores it exactly like any other detector, with true positives, false positives, and false negatives. A downvote is not special just because a human produced it. It is a noisy signal like any other. The only way to know how much to trust it is to run the same precision/recall check that any detector gets.
Explicit and implicit feedback
Explicit feedback is users answering a direct request (thumbs, stars, a form, an edit). Implicit feedback is inferred from behaviour (regeneration, rephrasing, abandonment, copying the answer).
Example: a downvote (explicit) catching only 5% of real failures while a regeneration (implicit) catches 36% is a reminder that "explicit" doesn't mean "reliable" -- the behavioural signal nobody had to deliberately give turned out to be the stronger detector here.