CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 8 of 12 · 2 min read

Answering incident questions in an interview

Your support agent started issuing wrong refunds. What do you do?

"Acknowledge and set severity; money moved makes it at least SEV2. Mitigate first: flip the refund tool kill switch so the agent can no longer issue refunds and route those cases to humans, accepting that legitimate refunds wait. Then check recent releases through the manifest; if a prompt or model change is the cause, roll back, then lift the switch. Preserve the traces, list the affected refunds from the audit log, fix them with the customers, and write a blameless post-mortem with an eval for the case, an alert on refund rate, and tighter approval limits." Follow-up: "How do you know the scope?" Trace and journal records tagged with release id and tenant (Level 9, module 9) plus the tool-call audit log.

How do you design alerting for an agent system?

"Symptom-based, with burn-rate alerts on success rate and quality proxies, plus agent-specific signals: cost per minute, steps per run, tool errors, escalation rate and eval or replay drift. Every page has an owner and a runbook, and I review pages monthly to remove noisy ones. In a simulation, burn rate cut false pages from 44 to 2 in a week at the cost of a few minutes' delay." Follow-up: "What about slow quality drift?" A scheduled replay or eval on production samples with a threshold and a trend alert (module 9, Level 9).

What goes in a post-mortem?

"Summary, impact, timeline with the incident clocks, root causes plural including contributing factors, how it was detected and why not sooner, what went well, and action items each with owner, due date and a kind: prevent, detect or mitigate. It is blameless, so it asks why the system allowed it." Follow-up: "How do you stop actions being ignored?" Track them like work items with owners and review them in a recurring meeting.

An agent leaks one tenant's data to another. What now?

"SEV1. Pause the affected tenants or the feature with a scoped switch, stop the leak path, preserve evidence, identify exactly which data went where using the audit log, notify according to contractual and legal duties with counsel, then fix the isolation bug (module 7) and add a regression test." Follow-up: "How would you have caught it earlier?" Canary tests with seeded cross-tenant probes and alerts on cross-tenant access denials.