Definitions

Definitions

The answer, step by step

1. Clarify. Who is the user? What is the job? What is the worst failure (wrong refund, leaked data)? What volume? Is a write action involved?
2. Targets. Put numbers on quality, p95, cost and availability. If the interviewer gives none, propose them and say so.
3. Baseline. One model call plus retrieval if the answer needs data. Estimate its latency, cost and quality.
4. Add boxes for a reason. Guardrails for risk, router for cost, cache for repeated queries, a loop only when one pass cannot do the job.
5. Failure modes. Timeouts, tool errors, bad plans, loops that never end, prompt injection, write actions that need approval.
6. Evaluate and monitor. Offline set, online sampling, traces, the three observability metrics above.
7. Trade-offs. Say what you gave up and what would make you reverse a choice.

Results

1. A cache moves the median far more than the tail: at 40% hits the median fell 37% but the p95 only 9%, because misses still take the slow path.
2. The loop dominates latency and cost once it exists, so bound it first.
3. Seed noise is small: ten runs of 50,000 requests gave p95 between 10.35 and 10.44 s.
4. The router trades quality for cost and only an evaluation can say whether the trade is acceptable.
5. Retries fix independent blips, not outages.

Design rules

• State targets before drawing boxes.
• Start from the baseline, and add a component only against a missed target.
• Estimate with distributions; do not add p95s.
• Bound every loop: maximum rounds, maximum cost, maximum time.
• Cache the expensive, repeatable steps, and check staleness.
• Put human approval in front of write actions, because they add risk that read-only steps do not.
• Design observability in: traces per step, cost per request, change failure rate.
• Say which numbers are assumptions and how the design changes if they are wrong.

Interview questions, with the follow-ups

Common mistakes

• Drawing the final architecture first without targets.
• Adding p95s or using only averages.
• Ignoring the loop's variable length.
• Forgetting evaluation: a router or small model without a quality check.
• Assuming independent failures.
• No cost number.
• Write actions without approval.
• Quoting made-up numbers as facts instead of labelled assumptions.

What the example does not do

• No measurement: every component number is invented.
• No correlation: outages, rate limits and shared dependencies are ignored.
• No quality model: quality is a fixed number per model; the router's errors and the small planner's losses are not modelled.
• No queueing: concurrency, load and rate limits can dominate real tail latency.
• No streaming: time to first token is not separated from total time.
• Optional experiment (I did not run this): replace each stage's p50 and p95 with values from your own project's traces and see which target fails first.

For the curious Every box buys something and costs something

Adding a router cut cost from $0.0090 to $0.0052 and p95 from 5.1 s to 4.2 s, but my assumed quality fell from 0.84 to 0.78, below a 0.80 target. The agent loop pushed p95 to 10.4 s against an 8 s target. A smaller planner model got p95 to 7.7 s in the simulation, which says only that it is worth testing, because the model does not capture planning quality. Availability treats failures as independent, so it is an upper bound; a provider outage hits every call at once.

What you must be able to say (one page)

• Use a fixed order. (1) Clarify the user, the job and the failure that matters. (2) Set targets: quality, p95 latency, cost per request, availability. (3) Draw the simplest baseline (one model call). (4) Add boxes only when a target is missed, naming the target each box serves. (5) Walk through failure modes and approval points. (6) Say how you would evaluate and monitor. (7) State the trade-offs you made. This order is my own.
• A common build-up order for the boxes: enhance context, add guardrails, add a model router and gateway, reduce latency with caches (exact and semantic), add agent patterns (loops, parallel branches, write actions), with monitoring and observability designed in from the start. Follow whichever order suits your application.
• Percentiles do not add. In my simulation of a fictional support agent with ten stages, the sum of the stage p95s was 12.6 s while the simulated request p95 was 10.4 s; the sum of medians was 5.2 s against a simulated 4.7 s. The back-of-envelope sum overstates the tail because not every stage is slow at once. In the other direction, retry timeouts push the real tail up: the model-only design simulated 4.6 s against a listed p95 of 4.5 s.
• Serial availability multiplies, retries raise it, and correlation breaks it. With three attempts allowed on each LLM call, request availability went from 98.6% (no retries) to 99.91% (one retry); further retries gained nothing because my model treats failures as independent. A provider outage is not independent: it hits every call at once. I did not model that, so treat these availabilities as upper bounds.
• Find the bottleneck by perturbation. Halving each stage's latency in turn, the planner LLM call saved 2.7 s of p95, the CRM tool 0.85 s, the answer model 0.70 s; guardrails under 0.1 s each. Optimise what the experiment shows, not what feels slow.
• Every box buys something and costs something. Router: cost fell from $0.0090 to $0.0052 per request and p95 from 5.1 s to 4.2 s, but my assumed quality fell from 0.84 to 0.78, under a 0.80 target. Agent loop: p95 rose to 10.4 s (target 8 s) and cost to $0.0125.
• Fixes are hypotheses. A smaller planner model brought the simulated p95 to 7.7 s and cost to $0.0063; capping the loop at two rounds alone only reached 9.1 s. My simulation does not model any loss in planning quality from the smaller model, so the answer is "test this", not "this works".
• Defend the numbers as assumptions. In an interview, state the figures you assume ("about 2 s median for a large-model call"), say they are assumptions, and show how the design changes if they are wrong (sensitivity).
• Two more points to say out loud. From the user's view, time to first token is much longer when an agent plans and calls tools before answering, so stream progress. Three useful operations metrics are mean time to detection, mean time to response and change failure rate. Resilience needs error handling, retries and fallbacks, with timeout and retry logic tuned to the latency requirement of the use case.
• Honest limits: the stage numbers, quality scores, cache hit rate and router share are assumptions; the code is arithmetic, not a measurement of any real system.