Part 8 of 12 · 2 min read
Answering feedback-loop questions in an interview
How do you use user feedback to improve an agent?
"I log signals with trace ids, sample and triage them by category, have experts label a sample to estimate real failure rates, fix the cause, add the failing case to the eval set, run the CI gate, then roll out gradually and watch guardrails. I treat thumbs as candidates for review, not as ground truth." Follow-up: "Why not optimise thumbs directly?" Users reward agreement as well as correctness; in the example's simulation selecting by thumbs dropped correctness from 84 to 59 to 66 percent, and at least one widely reported model update was rolled back for being too agreeable.
Your thumbs-up rate is 90 percent. Is the agent good?
"Not necessarily. Only a few percent of users give feedback, and if unhappy users are more likely to speak, the share is biased, which in the example's simulation underestimated satisfaction by 30 points. The opposite bias is also possible. I would sample outcomes independently and have experts label them." Follow-up: "Can you correct it?" Only if I know or can estimate the response probabilities, for example from a random survey; otherwise I report it as a trend, not a level.
How do you pick between two prompts using user votes?
"A planned A/B test with a pre-computed sample size (module 9), a guardrail set, and expert review of a sample, because choosing the best of many by noisy votes overstates the gain: 78 percent observed versus 73 percent true with 100 votes each." Follow-up: "How many variants should you test?" As few as practical, and confirm the winner on fresh traffic.
What implicit signals would you track for a chat agent?
"Regeneration, immediate rephrasing, abandonment after a response, copying or accepting the output, escalation to a human, and refusal-related drop-off. Each is noisy: in the example's simulation regeneration had 53 percent precision and 36 percent recall, so I calibrate each against expert labels." Follow-up: "What about agents that act?" Task completion, undo or correction by the user, and approval rejections (Level 8 module 8) are stronger signals than chat reactions.