Part 6 of 10 · 2 min read
Answering system-design questions in an interview
Design a customer-support agent platform for many companies.
"Requirements first: tenants, ticket volume, latency, and which actions the agent may take. I estimate about 100,000 tickets a day, 5.8 per second at peak, about 230 concurrent runs, so a dozen async workers; the real constraint is tokens, around 10 million per minute at peak against a provider limit, so I design for routing, caching and per-run caps. Architecture: authenticated tenant context, an async job API with idempotency, a fair per-tenant queue, durable workers, a model gateway with cascade and fallback, tenant-scoped data and namespaces, tools with least privilege and human approval for refunds, SSE progress, release manifests with an eval-gated canary, and tracing with cost per resolved ticket. I would launch with one tenant and human approval, then expand autonomy as evals support it." Follow-up: "What is the biggest risk?" Cross-tenant leakage through a model-controlled argument or shared cache (module 7), then prompt injection through ticket text (Level 10).
How do you keep cost per resolved task under control?
"Measure it per task from traces, cap each run, cache the stable prefix, trim context, and route with a cascade that has a verifier, all above a quality floor. In my estimate caching alone cut the support platform from about $322,000 to $138,000 a month, so cost is part of the architecture." Follow-up: "What if quality drops?" The eval gate and canary catch it; I look at cost and quality together, never cost alone.
Design a deep research agent.
"Runs are long and costly, so durable execution with checkpoints, cancellation, streamed progress and a hard budget per run, which my estimate put near $7 for 40 steps and three sub-agents. I start with a single agent loop plus an eval set, add parallel sub-agents only if they improve results, compact context, and verify citations and sources. Web content is untrusted, so injection defences and a source allow-list." Follow-up: "How do you evaluate it?" Outcome evals on source quality and faithfulness plus trajectory checks (Level 9 modules 3 and 6).
The provider rate limit is 2M tokens per minute. What do you do?
"Reduce demand first: caching, a smaller model for most tasks, shorter context, fewer steps. Then smooth demand with a queue and per-tenant fair share, add a second provider as fallback, request higher limits, and shed or delay low-priority work (modules 2, 5, 6)." Follow-up: "What if the limit is shared across tenants?" Per-tenant quotas so one tenant cannot consume it all.