Part 7 of 11 · 2 min read
Answering support-agent questions in an interview
Design a customer-support agent that issues refunds.
"Scope to refund, cancel, address change and status. A model classifies intent with a confidence, and a deterministic policy decides from there: low confidence, amounts above a limit, legal or fraud language and billing disputes go to a human with a handoff packet. Tools enforce preconditions (delivered, in the window, not already refunded, the customer's own order) and take idempotency keys. I launch in assist mode, measure error rates and calibration, then automate low-risk intents first." Follow-up: "How do you set the confidence threshold?" Sweep it on real traces against a cost model of escalation versus mistakes. In the example's simulation the cheapest threshold was not the most conservative.
Should the agent be fully autonomous?
"It depends on accuracy and the cost of a wrong action. In the example's simulation at 95 percent accuracy full autonomy was cheapest; at 75 percent it was the most expensive and made 55 wrong actions. So I make autonomy a per-intent setting driven by measured error and reversibility." Follow-up: "Which actions stay human?" Irreversible or high-value ones, legal and fraud cases, and anything the agent cannot explain.
What does a good escalation look like?
"A packet with the ticket, order, reason, what the agent tried and a suggested action, routed by skill and urgency, so the customer is not asked again. I track escalation rate, time to pickup and how often the human overrides the suggestion." Follow-up: "What if escalations overwhelm the team?" Look at the top escalation reasons and fix those intents or thresholds. Then add queue priorities and SLAs.
How would you design an operations agent that can restart services?
"Risk tiers: read-only runs freely, mutating actions need approval, destructive classes are blocked, unknown commands count as risky. Dry-run first, a blast-radius cap, a per-hour write limit, an audit trail and a kill switch." Follow-up: "How do you avoid approval fatigue?" Auto-approve low-blast-radius reversible actions after measured safety. Keep humans on the rest (Level 10 module 5).