Definitions

Definitions

One architecture

Results

1. The binding constraint was model tokens, not servers.
2. Caching and routing are not optimisations here; they decide whether the product is viable.
3. Most first drafts omit durability, isolation, safety and operations, exactly the topics that distinguish production designs.

Design rules (for the interview and for work)

• State requirements, volumes and targets before drawing anything; get the interviewer's numbers or state yours.
• Estimate load, concurrency, tokens and cost early; find the binding constraint.
• Draw the request path, then add durability, isolation, safety and operations.
• Choose async by default for agent runs; stream progress; make runs durable.
• Decide autonomy deliberately and start assistive (human approval) for risky actions.
• Put cost controls in the design: budget per run, caching, routing with a quality floor, quotas per tenant.
• List failure modes and the control for each.
• Plan evaluation and observability from day one: you cannot improve what you cannot measure.
• Say what you would build first and what you would defer, with the reason.
• Make trade-offs explicit: latency versus cost, autonomy versus risk, isolation versus cost, build versus buy.

Interview questions, with the follow-ups

Common mistakes

• Drawing boxes without requirements or numbers.
• Designing for servers when tokens are the bottleneck.
• Synchronous requests for minutes-long runs.
• Ignoring tenant isolation, or leaving it to "the model will not do that".
• No durability story: a worker kill restarts everything.
• Full autonomy on day one for money-moving actions.
• No cost cap, no eval, no rollback plan.
• No failure modes listed, or only provider outages.
• Over-engineering: Kubernetes, many agents and a vector database before a single working loop.
• Not saying what you would build first.

What the example does not do

• No system was built or load tested; the numbers are estimates from assumed inputs.
• Tokens per run assume full history re-send and ignore compaction, caching hit rates below 100 percent of prefixes, and provider-specific accounting (many providers count cached tokens differently against rate limits).
• Costs use illustrative prices, not a current price list, and ignore infrastructure, storage, engineering and human review costs.
• Little's law gives averages; real queues have variance, bursts and retries.
• The rubric is mine, not an official standard; real reviews depend on context.
• The two designs are sketches, not reference architectures; I did not validate them with a real team.
• Optional real experiment (I did not run this): take your own agent, measure real steps, tokens and run times from traces, replace the assumptions in design.py and compare the predicted cost and concurrency with a day of real traffic.

For the curious A design answer names its failure modes

Name a control for each: provider outage (breaker and fallback), worker kill (journal), noisy tenant (quota), runaway loop (cap), bad release (canary and rollback), prompt injection (Level 10), data leak (isolation and erasure), and cost spike (alerts and caps). All workload and price numbers are assumed round figures; the practice is the structure of the conversation.

What you must be able to say (one page)

• A system design answer is a structured conversation, not a diagram dump. A workable 45-minute shape: (1) clarify requirements and success metrics (5 min), (2) estimate load and cost (5), (3) sketch the architecture and walk one request through it (10), (4) go deep on the two or three riskiest parts (15), (5) failure modes, safety, operations (7), (6) trade-offs and what you would build first (3). Say your assumptions aloud; they are the point.
• Estimate first; it tells you where the problem is. For a support agent platform (assumed: 200 tenants, 500 tickets per tenant per day, peak 5 times the average, 6 model calls per ticket, 40 s per run): 100,000 runs per day, about 1.2 runs per second on average and 5.8 at peak; by Little's law (concurrent runs = arrival rate x time in system) about 231 concurrent runs at peak, so 12 async workers holding 20 runs each. Servers are not the problem. The model tokens are: with the history re-sent each step, about 28,350 input and 1,500 output tokens per run costs about $0.11 per run at the assumed large-model prices ($322,650 per month; $0.05 per run and $138,375 per month with prefix caching), and needs about 10.4 million tokens per minute at peak against an assumed 2 million limit (headroom 0.19). At 10 times the tenants the workers grow to 116 and token demand to 104 million per minute. For a research agent (assumed: 1,000 users x 5 runs per day, 40 steps, 3 parallel sub-agents, 15 minutes per run): 5,000 runs per day, 312 concurrent runs, 2.3 million input tokens per run, about $7.35 per run ($1.35 cached), 48.6 million tokens per minute at peak. All inputs are invented; the lesson is the method and the conclusion that cost and rate limits are architecture, driving routing, caching, context compaction (Level 7), budget caps and quotas (modules 5 and 7). These figures assume full-history resends and no compaction.
• The architecture is the level in one picture: clients with SSE; an API gateway that authenticates, derives the tenant context and issues job ids with idempotency (modules 1, 7); a fair per-tenant queue with retries and dead letters (2); durable workers with a journal, lease heartbeats and per-run budget caps (3, 5); a model gateway with routing, cascade, prefix cache, fallback and circuit breaker (5, 6); an event store for resumable streams (4); tenant-scoped data with row-level security and namespaces (7); tools with least privilege, sandboxes and human approval for risky actions (Level 10); a human queue with escalation; a release pipeline with manifests, pinned models, eval gate and canary (8, 9); observability with tenant-tagged traces and cost per resolved task (Level 9); and operations (alerts, runbooks, erasure registry; module 11).
• Support platform, key decisions and trade-offs. Latency target seconds to a couple of minutes, so async job API plus streaming progress. Most tickets are simple, so a cascade (cheap model, verifier, escalate) with a quality floor; sensitive actions (refunds) behind approvals and limits. Isolation per tenant is the biggest risk (identity from the session, namespaces, tenant-keyed caches, per-tenant quotas). Build first: one tenant, one ticket type, human approval on everything, tracing and evals, then widen.
• Research agent, key decisions and trade-offs. Runs last minutes, so durable execution, checkpoints and cancellation matter most (module 3), with streamed progress and budget caps (one run cost about $7 in my estimate). Fan-out sub-agents trade latency for cost and coordination complexity; context grows, so compaction and notes (Level 7). Quality is the hard part: citation checking, hallucination detection (Level 9 module 6), source allow-lists, and injection defences for web content (Level 10 module 1). Build first: a single-agent loop with a cap, tracing and an eval set; add sub-agents only if evals show they help.
• Always name the failure modes: provider outage (breaker and fallback, module 6), worker kill (journal, module 3), noisy tenant (quotas, module 7), runaway loop (cap, module 5), bad release (canary and rollback, module 8), prompt injection (Level 10), data leak (isolation and erasure), cost spike (alerts and caps), model deprecation (pinned snapshots and migration, module 9).
• Common cross-questions and one-line answers: build versus buy (buy the commodity, own the evals and the data boundary); sync versus async (async above a few seconds); one agent versus many (start with one; add agents when specialisation or parallelism pays and evals show it); SLA (define per ticket class; degrade gracefully); regional or regulated customers (stronger isolation tier, data residency, Level 10 module 10).
• Honest limits: these are my illustrative designs and assumed numbers, not a reference architecture for any real product; real interviews and real systems vary in scope.