CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 9 of 13 · 2 min read

Answering system-design questions in an interview

Design a customer-support agent that can look up orders and issue refunds.

"I would first ask the volume, the refund limit and what a wrong refund costs. Targets: p95 under 8 s, cost under 2 cents, 99.5 percent availability, quality at least 0.80 on an evaluation set. Baseline: retrieval plus one model call. Add input and output guardrails, a router to a small model for simple questions, a cache for repeated policy questions, and a bounded planner loop with the CRM tool. Refunds above a limit need human approval. I would instrument traces per step." Follow-up: "Where is the risk?" The write action and prompt injection through ticket text. Give the tool narrow permissions and require approval.

Your design has p95 of 10 s and the target is 8 s. What do you do?

"Measure, not guess: halve each stage in a simulation or from traces. In the example's example the planner call is the biggest lever (2.7 s). So a smaller planner model or fewer loop rounds. I would test the quality effect before shipping." Follow-up: "What if the quality drops?" Route only easy requests to the small planner, or stream partial results.

How do you estimate end-to-end latency?

"Sample each stage from a distribution fitted to its median and p95, compose in sequence, parallel and loops, and read the percentile of the total. Adding p95s overstates it (12.6 against 10.4 s in the example's example). Ignoring retry timeouts understates it." Follow-up: "What is the weakest assumption?" Independence between stages and between failures.

How do you make it reliable?

"Timeouts, bounded retries with backoff, fallbacks to a cheaper model or a cached answer, idempotent write actions, a circuit breaker for a failing provider, and monitoring of detection and response time." Follow-up: "Do retries always help?" No. They help with transient errors, not outages, and they add tail latency.