Interview questions
System Design Interview Questions
50 high-level and low-level system design questions, answered using a four-part structure: what the problem actually requires, how to architect it, the trade-offs the design accepts, and the specific failure mode it guards against. Covers both HLD (architecture-scale systems) and LLD (component- and class-level design) questions.
Jump to a level
- Level 1: Web building blocks — 1 question
- Level 2: Databases and storage — 3 questions
- Level 3: Caching and speed — 2 questions
- Level 4: Scaling data: replication and partitioning — 1 question
- Level 5: Transactions, consistency and consensus — 4 questions
- Level 6: Messaging, streams and pipelines — 5 questions
- Level 7: Architecture styles and services — 11 questions
- Level 8: Reliability and resilience — 5 questions
- Level 9: Operations, observability and security — 7 questions
- Level 10: Case studies I, core components — 1 question
- Level 11: Case studies II, products at scale — 7 questions
- Level 12: Case studies III, infrastructure and money — 2 questions
- Level 13: Practice, more designs and mock interviews — 1 question
Level 1: Web building blocks
Web building blocks — interview questions
Answer in your head first, then open it.
Design a Streaming Event Emitter for Agent UIs: Stream structured server-sent events (thought_delta, tool_start, tool_result, final_answer) over SSE or WebSockets.
Define: The emitter needs to push discrete, typed events to a connected client as the agent loop progresses, rather than the client waiting silently until the whole run finishes. Architect: I define a small, closed set of event types, each with its own payload shape, emitted at specific points in the agent loop, thought_delta as reasoning tokens stream, tool_start and tool_result around a tool call, final_answer at completion, serialized and pushed over an SSE connection, with the frontend rendering each event type with its own UI treatment rather than appending everything as plain text. Trade-offs: A small, typed set of event kinds is easy for a frontend to handle predictably, but it means any new kind of progress information the backend wants to surface later requires extending the shared event schema rather than just emitting arbitrary text. Fail-safe: A dropped connection mid-stream must not leave the frontend silently stuck; the emitter includes a periodic heartbeat event, and the frontend treats a missed heartbeat as a clear, surfaced connection error rather than an indefinitely frozen UI.
Level 2: Databases and storage
Databases and storage — interview questions
Answer in your head first, then open it.
Design a Distributed Agent Memory Service: Store, retrieve, update, and expire short-term session state and long-term semantic/episodic user memory at scale.
Define: This service needs two different storage shapes behind one interface: fast, short-lived session state and slower, durable long-term memory, both scoped per user and both needing to scale to many concurrent users. Architect: I back short-term session state with a low-latency key-value store with a TTL for automatic expiry, back long-term memory with a combination of a structured database for facts and a vector index for semantic recall, and expose both through one memory-service API so a calling agent doesn't need to know which backing store a given read or write actually hits. Trade-offs: Splitting storage by access pattern, a fast TTL-based cache for short-term and durable indexed storage for long-term, matches each workload to the right technology, but it means the service has to keep two different consistency and failure models coherent behind one interface. Fail-safe: A session TTL expiring mid-task and silently dropping state the agent still needed is a concrete risk; I extend the TTL on every active write rather than setting it once at session start, so an actively running session can't expire out from under itself.
Design a Hierarchical Memory Store (MemoryManager): Define interfaces for working memory, vector episodic memory, and structured key-value user facts with TTL and conflict resolution.
Define: This manager needs one interface in front of three different storage shapes, fast in-context working memory, a vector store for episodic recall, and a key-value store for structured facts, each with its own consistency needs. Architect: I expose a single MemoryManager interface with read and write methods parameterized by memory kind, routing working-memory operations to the active request's own state, episodic operations to the vector store with embedding and similarity search handled internally, and fact operations to the key-value store with a TTL on each entry and an explicit conflict-resolution rule, preferring the most recent write by default, applied whenever two writes target the same key. Trade-offs: One unified interface hides real differences in consistency and latency between the three backing stores, which simplifies calling code but means a caller can't easily reason about which operations are fast and which are not without reading the implementation. Fail-safe: Silently overwriting a fact on conflict can discard something still correct; conflicting writes are logged even as the most-recent-wins rule resolves them automatically, so a pattern of frequent conflicts surfaces as a data-quality signal rather than disappearing.
Design a Large Payload Offloader (Artifact Store): Intercept massive tool outputs (e.g., 5MB JSON/CSV), store them in blob/memory storage, and pass a lightweight reference handle to the LLM.
Define: A tool output far too large to put in the model's context needs to be stored outside the prompt entirely, with only a small reference passed through so the model can still work with it indirectly. Architect: The offloader sits between a tool's raw output and the agent loop, checking the output's size, and for anything over a configured threshold, writes it to blob storage and returns a lightweight artifact handle, an ID plus a short preview or schema summary, in place of the raw content; any downstream tool that needs the actual data fetches it directly from blob storage using that handle rather than it ever passing back through the model's context. Trade-offs: Offloading keeps the context window free regardless of how large a tool's raw output gets, but it means every downstream consumer of that output has to know how to resolve an artifact handle rather than just reading content directly from the prompt. Fail-safe: A downstream tool that silently falls back to requesting the full raw content inline when handle resolution fails can blow the context budget without warning; that fallback is made to fail loudly and visibly instead, forcing the handle-based path to actually work.
Level 3: Caching and speed
Caching and speed — interview questions
Answer in your head first, then open it.
Design a Real-Time Voice AI Agent: Achieve sub-500ms end-to-end latency with interruption handling (barge-in), streaming STT → LLM → TTS, and background tool calls.
Define: The hard constraint here is round-trip latency under 500 milliseconds across three chained AI stages, plus the ability for the user's voice to interrupt and cancel whatever the system is currently doing. Architect: I stream audio into speech-to-text incrementally rather than waiting for a full utterance, start the LLM call as soon as a stable partial transcript is available, stream the LLM's output tokens directly into text-to-speech rather than waiting for a complete response, and run any tool call that doesn't block the spoken reply in the background, with a short filler response covering the gap; a voice-activity detector watching the input stream fires a cancellation signal into all three stages the instant it detects the user speaking again. Trade-offs: Streaming every stage and reacting to partial transcripts cuts latency dramatically but risks acting on a transcript the user hasn't actually finished saying, so the barge-in detector has to be tuned carefully against false triggers from background noise. Fail-safe: A cancelled mid-flight LLM or tool call that keeps running and later produces a now-irrelevant spoken response is the specific risk; every stage checks a live cancellation token before producing output, so a barge-in event immediately silences anything still in flight rather than letting a stale response play late.
Design a Semantic & Tool-Result Caching Layer: Cache repetitive LLM reasoning steps and expensive external API calls with smart cache invalidation.
Define: Two different things need caching here: LLM reasoning that's semantically, not just textually, repeated, and external tool results that are expensive to re-fetch but can go stale. Architect: For LLM reasoning, I embed incoming requests and match against a cache of previously embedded requests and their outputs using a similarity threshold rather than requiring an exact match; for tool results, I cache by the exact call signature with an explicit time-to-live appropriate to how fast that specific data source actually changes, invalidating early on any explicit signal that the underlying data changed. Trade-offs: Semantic caching catches far more repeat traffic than exact-match caching, but too loose a similarity threshold risks serving a cached answer for a query that wasn't actually asking the same thing, which is a correctness risk, not just a performance one. Fail-safe: I set the similarity threshold conservatively and test it specifically against known near-duplicate-but-different query pairs, and for tool-result caching, I always expose a way to force a fresh fetch, bypassing the cache, for any case where staleness is unacceptable.
Level 4: Scaling data: replication and partitioning
Scaling data: replication and partitioning — interview questions
Answer in your head first, then open it.
Design an Agentic RAG Pipeline for 100M+ Documents: Combine hybrid vector/keyword search, query decomposition, document grading, and strict RBAC permissions.
Define: At 100M+ documents, the bottleneck shifts from whether retrieval works at all to whether it stays fast and correctly permission-filtered at this scale, which changes several design choices from a small-corpus RAG system. Architect: I partition the vector and keyword indexes by a sharding key, such as document source or tenant, so a query only has to search the relevant shards rather than the full corpus, apply RBAC filtering as a hard pre-filter at the index layer before ranking rather than after, and keep the hybrid fusion and query-decomposition logic from standard agentic RAG unchanged, since those don't change with scale, only the index architecture underneath them does. Trade-offs: Sharding by source or tenant keeps individual queries fast at this scale, but a query that genuinely needs to search across many shards, a cross-tenant search, loses that speed advantage and has to fan out to every relevant shard. Fail-safe: RBAC filtering applied only after ranking risks a wrong-permission document briefly existing inside the ranked candidate set; I filter by permission before ranking, at the index layer, so an unauthorized document is never even a candidate.
Level 5: Transactions, consistency and consensus
Transactions, consistency and consensus — interview questions
Answer in your head first, then open it.
Design a Multi-Turn User Session & Concurrency Lock: Prevent race conditions when a user sends two messages simultaneously to the same active agent thread.
Define: Two messages arriving concurrently for the same session thread must not both be processed as if they were the only one, since that risks the session's state being read and written out of order. Architect: I acquire an explicit, short-lived lock keyed by session ID before any step that reads and then writes session state, so a second concurrent message against the same session either queues behind the first or is rejected immediately with a clear still-processing response, rather than the two interleaving their reads and writes. Trade-offs: Locking session state prevents corruption from concurrent access, but it means the second message has to wait or fail fast rather than being processed in parallel, which needs to be a deliberate, visible behavior to the user, not a silent delay. Fail-safe: A lock that's never released because the holder crashed would permanently stall that session; every lock carries a short expiry so a crashed holder's lock is automatically released rather than requiring manual intervention.
Design an Idempotent Tool Executor: Prevent duplicate side effects (e.g., charging a card twice) using idempotency keys, deduplication stores, and state checks.
Define: The executor needs to guarantee that a tool call with a real-world side effect produces that effect exactly once, even if the same logical call is attempted more than once. Architect: Every mutating call is assigned an idempotency key generated once per logical action; before executing, the executor checks a deduplication store for that key, returning the previously recorded result immediately if found, and only if the key is new does it proceed to the real side effect and then record the result against that key before returning. Trade-offs: Checking and recording against a deduplication store adds a small amount of latency and storage to every mutating call, but without it, any retry after an ambiguous or lost response risks applying the side effect more than once. Fail-safe: A race between two near-simultaneous calls with the same key both passing the dedup check before either writes its result is the specific risk; the check-and-reserve has to be one atomic operation against the store, not a separate read followed by a separate write.
Design a Parallel Tool Execution Orchestrator: Execute independent tool calls concurrently (asyncio.gather / worker pools) while preserving deterministic result ordering.
Define: When the model requests several tool calls with no dependency between them, the orchestrator should run them concurrently for speed, but return their results in a stable, predictable order regardless of which one actually finishes first. Architect: I dispatch all independent calls concurrently, using an async gather or a worker pool, but collect results into a list indexed by the original request order rather than the completion order, so the model always sees results back in the same order it issued the calls, even though they executed out of order internally. Trade-offs: Running calls concurrently cuts total latency substantially when several are independent, but it requires validating up front that none of the batched calls actually depend on another's result, since forcing dependent calls into this path would produce wrong results. Fail-safe: One slow call in a batch dragging down the whole batch's return time is a real risk; each call in the batch gets its own timeout so one hung dependency doesn't block the others from completing and being returned as a partial result with a clear per-call status.
Design a Loop & Deadlock Detector: Detect repetitive tool-call patterns (exact or semantic duplicates) and inject corrective system prompts or terminate gracefully.
Define: This detector needs to notice when an agent is repeating the same or a semantically equivalent action without making progress, before that repetition burns through the full iteration or token budget. Architect: I keep a rolling window of recent tool calls, their names and arguments, compare each new call against that window for an exact match and, separately, an embedding-similarity match against recent reasoning text for a semantic repeat, and once a repeat is detected, inject a corrective system message naming the detected loop and asking for a different approach, escalating to a graceful termination with a clear stuck status if the loop continues after the correction. Trade-offs: Semantic-similarity detection catches rephrased repeats that an exact-match check would miss, but it costs an embedding comparison on every step and needs a carefully tuned similarity threshold to avoid flagging genuinely different but topically similar actions as a loop. Fail-safe: Terminating silently when a loop is detected hides the failure from whoever's reviewing the run; a detected loop always produces an explicit, logged terminated-due-to-detected-repetition status rather than just stopping with no explanation.
Level 6: Messaging, streams and pipelines
Messaging, streams and pipelines — interview questions
Answer in your head first, then open it.
Design a Human-in-the-Loop (HITL) Approval Architecture: Pause an active multi-step agent execution for hours or days waiting for human approval via Slack/Email, then resume seamlessly.
Define: The core requirement is that an in-progress execution can be suspended for an arbitrarily long, unpredictable duration without tying up any compute, and resume exactly where it left off once a human responds. Architect: I durably persist the full execution state to storage at the pause point, publish an approval request to a message queue that fans out to Slack and email, and have the approval response, arriving as a webhook, look up the persisted state by a correlation ID and push a resume event back onto the execution queue, rather than keeping any worker thread alive and waiting. Trade-offs: Durable persistence plus an event-driven resume costs more engineering than a simple in-memory sleep, but an in-memory wait can't survive a worker restart or scale past a handful of concurrent waiting executions, and this system needs to support many pending approvals for days at a time. Fail-safe: A second approval click, or a delayed webhook retry, resuming the same paused execution twice is the real risk; I key the resume action with an idempotency token tied to that specific pause point, so a duplicate approval event is a no-op rather than a double-resume.
Design an Asynchronous Background Agent Engine: Handle long-running research or coding tasks (10–60 minutes) using durable execution (e.g., Temporal/Celery), queues, and webhooks.
Define: This needs a task that can run for up to an hour, surviving worker crashes and restarts along the way, with the originating client notified only once it's actually done rather than holding a connection open. Architect: I submit the task to a durable execution engine or a job queue backed by persistent storage, breaking the task into checkpointed sub-steps so a crash resumes from the last completed step rather than from scratch, and fire a webhook to the client's registered callback URL once the task reaches a terminal state, with the client also able to poll a status endpoint as a fallback. Trade-offs: Durable execution with checkpointing adds real infrastructure, a queue, a state store, retry logic, compared to a simple background thread, but a simple thread has no survivable state if the worker process dies thirty minutes into an hour-long task. Fail-safe: A webhook delivery that fails because the client's endpoint is briefly down must not be treated as the task itself failing; I retry webhook delivery independently with backoff and keep the task's own completed result durably stored, so a client can always poll and recover the result even if every webhook attempt failed.
Design a Multi-Agent Handoff & Message Bus: Define typed message envelopes, shared scratchpads, and routing interfaces between a Supervisor and Worker agents.
Define: This needs a consistent way for a supervisor and multiple workers to exchange messages and shared state without every pair needing custom glue code. Architect: I define a typed message envelope with a sender, a recipient, a message type, and a structured payload, published onto a shared bus; a shared scratchpad is a structured object keyed by task ID that any agent can read and write specific fields of, with the bus itself responsible only for routing envelopes to the right recipient, not interpreting their contents. Trade-offs: A generic typed envelope and bus keeps the routing layer simple and reusable across any pair of agents, but it means payload validation has to happen at each agent's own boundary rather than being enforced centrally by the bus. Fail-safe: An envelope addressed to a worker that's currently unavailable, crashed or not yet started, must not simply vanish; the bus persists undelivered envelopes and redelivers them once the recipient becomes available, rather than dropping anything sent to a momentarily-unreachable agent.
Design a Plan-and-Execute Task Scheduler: Model a dynamic Directed Acyclic Graph (DAG) of subtasks where an agent can add, retry, or skip tasks mid-execution.
Define: Unlike a static DAG fixed at the start, this scheduler needs to let the DAG's own shape change while it's running, since the agent may discover a step is unnecessary or that a new step is needed partway through. Architect: I represent the plan as a mutable DAG of task nodes with explicit dependency edges, a scheduler that dispatches any task whose dependencies have all completed, and explicit operations to add a new task with its dependencies, mark a task skipped, propagating that status to anything depending only on it, or retry a failed task up to a bounded count, with every mutation to the DAG itself logged for later replay. Trade-offs: Allowing the DAG to mutate mid-execution gives the flexibility real agent planning needs, but it makes the scheduler considerably harder to reason about and test than a DAG whose shape is fixed before execution starts. Fail-safe: An added task that accidentally creates a cycle would deadlock the scheduler; every mutation is validated against the current graph for cycles before being accepted, rejecting any change that would make the DAG un-schedulable.
Design a Long-Running Async Tool Poller: Handle tools that return a 202 Accepted job ID by polling or awaiting callbacks without blocking worker threads.
Define: A tool that kicks off a long job and returns immediately needs its eventual result tracked and surfaced back to the agent loop without tying up a worker thread for the whole duration. Architect: On a 202 response, the poller records the job ID against the originating request and returns control immediately; a separate lightweight scheduler polls the job's status endpoint on a backoff interval, or waits on a webhook callback where the external system supports one, and only once the job reaches a terminal state does it push the result back into the agent's execution queue to resume that paused step. Trade-offs: Polling is simple to implement against any API but wastes some requests checking a job that isn't done yet; a webhook callback is more efficient but only works if the external system actually supports pushing a notification, so many real integrations need to support both. Fail-safe: A job that never reaches a terminal state, stuck indefinitely on the external system's side, must not leave the agent waiting forever; every tracked job gets a maximum wait time after which it's marked as timed out and surfaced to the agent as a failure.
Level 7: Architecture styles and services
Architecture styles and services — interview questions
Answer in your head first, then open it.
Design a Multi-Tenant Enterprise Agent Platform: Support thousands of organizations creating custom agents with isolated data, tool permissions, and token budgets.
Define: This is a platform-design problem: many independent organizations must each get their own isolated agents, data, and spend limits on one shared piece of infrastructure, without any tenant able to see or affect another's. Architect: Every request carries a tenant ID that scopes it end to end: a per-tenant namespace in the data store, a per-tenant tool-permission set loaded at session start, and a per-tenant token-budget counter checked before every model call; the orchestration layer itself stays stateless and generic, with tenant-specific configuration loaded as data rather than baked into code paths. Trade-offs: Scoping everything by tenant ID at the data layer, rather than relying on application code to remember to filter, costs more upfront schema and access-control design, but it's the only approach that actually prevents a missed filter in one code path from leaking another tenant's data. Fail-safe: A tenant whose token budget silently got confused with another tenant's is the specific failure to guard against; I enforce the budget check as a hard, atomic decrement against that tenant's own counter at the point of the model call, not a periodic reconciliation job that could let a burst through before catching up.
Design a Multi-Agent Supervisor & Worker System: Route complex user queries across 15+ specialized sub-agents without blowing up latency or losing shared context.
Define: The challenge is routing a request to the right subset of many specialized agents while keeping total latency bounded and making sure context gathered by one agent is actually available to the ones that need it next. Architect: A lightweight classifier-style supervisor, not a full reasoning pass, decides which of the 15+ workers are actually relevant to a given request, dispatches to only those in parallel where they're independent, and maintains a shared, structured context object that each worker reads from and writes specific fields back into, rather than passing full conversation transcripts between every pair. Trade-offs: Routing to a small relevant subset instead of consulting all 15+ workers keeps latency bounded, but a classifier that's wrong about which workers are relevant will silently under- or over-route, so its accuracy has to be measured the same way tool-selection accuracy is measured anywhere else. Fail-safe: An unbounded fan-out, where the supervisor decides many workers are relevant to a single request, can make latency unpredictable; I cap the maximum number of workers dispatched per request and require the supervisor to rank and pick the top few rather than invoke every plausible match.
Design a Universal Tool & MCP Gateway: Expose 500+ internal company APIs to agents with dynamic tool discovery, OAuth delegation, and rate limiting.
Define: This is an API-gateway design problem at agent scale: hundreds of internal APIs need to be exposed through one consistent interface, with per-caller auth delegation and rate limits, rather than every agent integrating with every API directly. Architect: The gateway maintains a registry of tool definitions backing onto the real internal APIs, performs OAuth token exchange so the agent never holds a raw internal credential directly, enforces a token bucket rate limiter per API and per calling agent, and exposes a discovery endpoint that returns only the subset of the 500+ tools relevant to the current request. Trade-offs: Centralizing all internal API access through one gateway makes auth, rate limiting, and auditing consistent everywhere, but it also makes the gateway itself a chokepoint whose own availability now gates every single tool call across the company. Fail-safe: The gateway becoming a single point of failure is the structural risk; I run it as a horizontally scaled, stateless fleet behind a load balancer with no single instance holding unique state, so losing any one instance doesn't take down tool access entirely.
Design the Core Agent Execution Loop (AgentRunner): Implement state transitions, max-iteration bounds, stop conditions, and step callbacks cleanly in code.
Define: The AgentRunner is the single piece of code responsible for driving the observe-decide-act cycle, enforcing its own hard limits regardless of what the model's reasoning decides. Architect: I model the runner as an explicit loop over a typed state object, with each iteration calling the model, dispatching any returned tool call, writing the result back into state, and checking a stop condition, an explicit final-answer signal or a hard iteration cap, before continuing; step callbacks fire after each iteration so external code, like a UI or a tracer, can observe progress without being woven into the loop's own control flow. Trade-offs: Keeping the loop's control flow fully explicit in one place, rather than spread across callback chains or a framework's internals, makes it easy to reason about and test, but it means re-implementing anything a heavier framework would have provided, like multi-agent handoff, if the runner later needs it. Fail-safe: The iteration cap is enforced by the runner itself, never by the model deciding on its own that it's done enough; hitting the cap always produces an explicit incomplete, hit-the-limit result rather than silently returning whatever partial state existed when it stopped.
Design a Graph-Based State Machine (StateGraph): Model nodes, conditional edges, cycles, and typed state reducers (like LangGraph) using OOP/functional patterns.
Define: A StateGraph needs to let a developer declare nodes and the conditional edges between them, including cycles, and have the typed state updated predictably as execution moves through the graph. Architect: Each node is a function taking the current typed state and returning a partial update; a reducer merges that partial update into the full state using an explicit merge rule per field, rather than a blind overwrite; edges are declared as conditions evaluated against the current state to decide which node runs next, and a cycle is simply an edge pointing back to an already-visited node, with the graph requiring every cycle to have at least one path to a terminal node reachable from it. Trade-offs: Typed reducers per field make state updates predictable and easy to debug compared to letting any node mutate any part of state freely, but defining a reducer for every field is more upfront design work than just allowing arbitrary mutation. Fail-safe: An unbounded cycle with no reachable terminal node is a structural bug in the graph definition itself; I validate at graph-construction time that every cycle has a path out, rather than discovering an infinite loop only at runtime.
Design a Context Window Manager (ContextPruner): Implement token counting, sliding windows, tool-output truncation, and rolling summarization strategies.
Define: This component's job is keeping the prompt within a token budget as a trajectory grows, deciding what gets dropped, truncated, or summarized without discarding something the agent still needs. Architect: I count tokens incrementally as each new message or tool output is appended, rather than recomputing the whole prompt every time, apply a sliding window that keeps the most recent raw turns intact, truncate individual oversized tool outputs to a configured length with a note that truncation happened, and run a periodic summarization pass that folds turns falling out of the sliding window into a running summary rather than dropping them outright. Trade-offs: Summarizing dropped turns instead of discarding them preserves more information per token, but it costs an extra model call to generate the summary and risks the summary quietly losing a specific detail in the compression. Fail-safe: A small set of explicitly protected facts, the original goal, hard constraints, is kept outside the sliding window and summarization process entirely, in a fixed slot that's never pruned, so the manager can be aggressive everywhere else without risking those.
Design a Reflection / Critic Loop Controller: Structure the interface between a Generator, an Evaluator/Critic, and a Retry budget.
Define: This controller coordinates three distinct roles, something that produces an answer, something that judges it, and a bounded budget for how many times that cycle can repeat, keeping them cleanly separated. Architect: I define a Generator interface that produces a candidate given the task and any prior critique, a Critic interface that scores a candidate and returns structured feedback, and a Controller that loops: generate, critique, and if the critique fails a threshold and the retry budget isn't exhausted, generate again with the new critique as input, stopping either on a passing score or an exhausted budget, whichever comes first. Trade-offs: Keeping Generator and Critic as separate, swappable interfaces lets either be improved or replaced independently, but it costs at least double the calls of a single-pass generation, and a critic with its own blind spots can wrongly pass or fail a candidate. Fail-safe: Without a hard retry budget, reflection can oscillate indefinitely between two corrected answers; the controller always tracks the best-scoring candidate seen across all attempts and returns that one when the budget runs out, rather than trusting whichever attempt happened to run last.
Design a Tool Registry & Schema Generator: Auto-generate JSON schemas from typed functions/docstrings, validate arguments, and support dynamic tool filtering.
Define: This registry needs to turn ordinary typed functions into the JSON schema format an LLM tool-calling API expects, automatically, so a developer writes one function and gets both the callable implementation and its schema for free. Architect: I inspect each registered function's type annotations and docstring at registration time to generate its JSON schema automatically, validate every incoming tool-call argument against that generated schema before invoking the real function, and expose a filtering method that returns only the subset of registered tools matching a given criterion, a category tag or a retrieval query, so a caller with hundreds of tools can request just the relevant few. Trade-offs: Auto-generating schemas from code keeps the schema and the implementation from drifting apart, since there's only one source of truth, but it means the schema's quality is only as good as the function's own type hints and docstring, which still need to be written carefully. Fail-safe: A function whose type hints don't actually match its runtime behavior produces a schema that lies about what's valid; I validate a sample of real calls against the generated schema in a test pass before trusting it in production, rather than assuming generation alone guarantees correctness.
Design an MCP (Model Context Protocol) Client/Server Wrapper: Implement resource discovery, prompt templates, and standardized tool invocation over stdio/SSE.
Define: This wrapper needs to speak MCP on both sides, exposing a server's tools, resources, and prompt templates to a client over a standard transport, whether that's a local stdio pipe or a networked SSE connection. Architect: The server side registers its tools, resources, and prompt templates against the protocol's discovery methods, responding to a client's discovery request with their schemas; the client side connects over the configured transport, calls discovery once per session to learn what's available, and translates each subsequent tool invocation into the protocol's standard request-response shape, so the calling agent code only ever sees ordinary typed tool calls regardless of which transport is underneath. Trade-offs: Supporting both stdio and SSE transports behind one client interface adds an abstraction layer the code has to maintain, but it means the same agent code works against a local MCP server during development and a networked one in production without changes. Fail-safe: A server that changes its tool schemas without the client re-running discovery would leave the client calling against a stale schema; the client re-runs discovery on any invocation failure that looks schema-related, rather than caching discovery results indefinitely.
Design a Structured Output Parser & Auto-Repair Loop: Validate LLM JSON responses against Pydantic/JSON schemas and feed validation errors back for self-correction.
Define: This component needs to turn the model's raw text response into a validated, typed object, and when validation fails, give the model a chance to fix its own output rather than failing the whole call outright. Architect: I parse the raw response as JSON and validate it against the target schema; on a validation failure, I construct a correction prompt containing the specific validation error, not just invalid JSON, and re-call the model with that correction for a bounded number of attempts, only raising a hard failure to the caller once the retry budget is exhausted. Trade-offs: Giving the model specific validation errors to self-correct recovers from many minor formatting mistakes automatically, but each retry costs an additional model call, so a persistently malformed response still eventually has to fail rather than retry indefinitely. Fail-safe: A response that's valid JSON and matches the schema's shape but has nonsensical values can still pass this layer; schema validation guarantees shape, not semantic correctness, so it has to be paired with separate value-level validation for anything where that distinction actually matters.
Design a Modular Guardrail Pipeline: Chain input validators, PII redactors, and output policy checks using the Chain-of-Responsibility or Middleware pattern.
Define: This pipeline needs to run a sequence of independent checks, each able to inspect, modify, or reject content, without hardcoding which specific checks run in what order into the core request-handling code. Architect: I implement each guardrail as a middleware function taking the content and a next callable, able to inspect or transform the content, call next to continue the chain, or short-circuit by returning a rejection directly; the pipeline itself is just an ordered list of these middleware functions composed together, so adding, removing, or reordering a guardrail is a configuration change, not a code change to the core handler. Trade-offs: The middleware pattern makes guardrails independently testable and easy to reorder, but a long chain of sequential checks adds their combined latency to every request, so performance-sensitive checks may need to run in parallel rather than strictly in sequence. Fail-safe: A guardrail that throws an unexpected exception must not silently let content through the pipeline unchecked; any unhandled failure in a guardrail step defaults to rejecting the content rather than treating an error as an implicit pass.
Level 8: Reliability and resilience
Reliability and resilience — interview questions
Answer in your head first, then open it.
Design a Model Routing & Fallback Gateway: Route requests dynamically between small local models and frontier models based on task complexity, latency SLAs, and provider outages.
Define: This gateway needs to pick, per request, which model tier to call based on how hard the task looks, how tight the latency budget is, and whether the preferred provider is currently healthy. Architect: A lightweight complexity classifier or a rule set tags each incoming request, the gateway maintains live health signals per provider from recent error and latency rates, and routing combines both: a simple request goes to a small local model by default, but any provider marked unhealthy is skipped in favor of the next ranked fallback regardless of which tier it belongs to. Trade-offs: Dynamic routing based on live health signals adapts automatically to an outage, but it adds a layer of indirection that makes debugging which model actually answered a given request harder unless every routing decision is logged explicitly. Fail-safe: Routing a request to a degraded fallback silently, with no record that it happened, hides a real reliability event; I log the routing decision and its reason for every single request, not just on fallback, so frequent fallback activation is visible rather than invisible.
Design a Checkpointer & State Persistence Interface: Serialize, version, save, and replay agent execution snapshots after every step for crash recovery and time-travel debugging.
Define: The checkpointer needs to capture a complete, versioned snapshot of execution state at each step, cheaply enough to do it after every single step, and be able to restore or replay from any saved snapshot. Architect: I define a checkpointer interface with save, given the current state and a step identifier, and load, given a step identifier, returning that saved state, backed by a durable store keyed by run ID and step number; each snapshot is tagged with a schema version so a checkpoint saved under an older state shape can still be loaded and migrated rather than silently failing to deserialize. Trade-offs: Checkpointing after every single step gives the finest-grained recovery and the most complete time-travel debugging, but it adds a write to durable storage on every step, so a very high-throughput loop may need to checkpoint at a coarser interval instead. Fail-safe: A checkpoint written in an old schema version that a newer runner can't parse is a real risk after any state-shape change; the load path always checks the stored version and runs an explicit migration step before handing the state to current code.
Design a Circuit Breaker & Retry Wrapper for Tools: Handle external API timeouts, exponential backoff, jitter, and structured error formatting back to the LLM.
Define: This wrapper needs to retry a transient external failure sensibly, but stop calling an API that's clearly down, rather than hammering it, and give the model an error it can actually act on. Architect: I wrap each external call with exponential backoff and jitter for a bounded number of retries on transient errors, track a rolling failure rate per external dependency, and trip a circuit breaker that short-circuits further calls to a clearly failing dependency for a cooldown period rather than letting every caller keep timing out against it; any failure that reaches the model comes back as a structured error with a reason code and a suggested next step, not a bare exception. Trade-offs: Tripping a circuit breaker protects the system from piling retries against a dependency that's actually down, but it also means a dependency that's recovering gets no traffic at all until the cooldown expires, even if it would have succeeded sooner. Fail-safe: A breaker that never resets would permanently disable a recovered dependency; it periodically allows a small number of trial calls through during the cooldown and closes again automatically once those succeed, rather than requiring a manual reset.
Design a Human-in-the-Loop Interrupt Mechanism: Implement an exception/signal-based pause mechanism that saves execution state and resumes upon receiving a signed approval token.
Define: This mechanism needs to cleanly halt execution at a defined point, persist enough state to resume later, and only resume on a cryptographically verifiable approval, not just any incoming signal. Architect: An interrupt point raises a dedicated pause exception that the runner catches, serializing the current state to durable storage keyed by a correlation ID before returning control; resumption requires a signed token proving the approval actually came from an authorized approver, which the runner verifies before loading the saved state and continuing execution from that exact point. Trade-offs: Requiring a signed token for resumption adds a verification step and key-management overhead compared to trusting any resume request, but trusting an unsigned resume signal would let anyone who can reach the resume endpoint approve a sensitive action. Fail-safe: A forged or replayed resume request is the specific attack this guards against; each approval token is single-use and tied to the specific correlation ID, so even a captured valid token can't be replayed to resume a different paused execution.
Design a Resilient LLM Provider Client: Implement unified interfaces across OpenAI/Anthropic/Gemini with automatic fallback, rate-limit token buckets, and cost tracking.
Define: This client needs to expose one consistent interface to the rest of the application regardless of which underlying provider actually serves a given call, while handling each provider's own rate limits and failures transparently. Architect: I define a provider-agnostic request and response shape, with a thin adapter per provider translating to and from that provider's own API format, a token bucket rate limiter per provider and per API key to stay under its limits, a ranked fallback list tried in order when the primary provider errors or is rate-limited, and cost tracked per call using each provider's own published pricing applied to the actual token counts returned. Trade-offs: A unified interface across three different providers' APIs, each with their own quirks, is real translation-layer work to build and maintain as each provider's API evolves, but it's what lets the rest of the application stay provider-agnostic. Fail-safe: Falling back to a different provider silently, with no record of it happening, hides a real reliability event; every fallback activation is logged explicitly with the reason, so a pattern of frequent fallback use is visible to whoever's monitoring the system.
Level 9: Operations, observability and security
Operations, observability and security — interview questions
Answer in your head first, then open it.
Design a Sandboxed Code-Execution Environment for Agents: Allow coding/data-analysis agents to run untrusted Python/Bash scripts securely using ephemeral containers or microVMs (e.g., Firecracker).
Define: The requirement is running code the agent itself wrote, which can't be trusted, in an environment that fully contains whatever that code tries to do. Architect: Each execution gets a fresh microVM or container with no persistent state, no network access beyond an explicit allowlist, a hard CPU, memory, and wall-clock limit, and the sandbox is torn down completely after the run rather than reused for the next one, so nothing from one execution can leak into the next. Trade-offs: Spinning up a fresh microVM per execution gives the strongest isolation guarantee but costs real startup latency compared to reusing a warm container, so a system with high execution volume needs a pool of pre-warmed, disposable sandboxes rather than cold-starting every call. Fail-safe: A sandbox escape, where isolation turns out weaker than assumed, is the risk that actually matters; I treat the isolation boundary as something to independently security-test and patch on an ongoing basis, not a one-time setup that stays safe forever.
Design an Agent Observability & Distributed Tracing System: Capture nested LLM calls, tool latencies, token costs, and state transitions across multi-agent hops.
Define: A single user request can fan out across many agents and model calls, so the tracing system needs to reconstruct that whole nested structure, not just log isolated events. Architect: Every call, model or tool, is instrumented as a span tagged with a shared trace ID and a parent span ID, so spans naturally nest into a tree reflecting the real call structure across every agent hop, with each span carrying its latency, token count, and cost, sent to a backend built for this shape of hierarchical trace data. Trade-offs: Capturing full nested traces on every request gives complete visibility, but it adds real overhead in storage and some latency, so very high-volume systems often need to sample full traces while always capturing summary metrics. Fail-safe: Sampling traces risks missing exactly the rare failure that most needed investigating; any request that errors or gets flagged by an evaluation check always gets a full trace captured regardless of the sampling rate applied to ordinary traffic.
Design an Offline & Online Agent Evaluation Pipeline: Run deterministic regression tests, trajectory comparisons, and LLM-as-a-Judge scoring on every prompt or model change.
Define: Every change to a prompt or underlying model needs to be checked against a fixed dataset before shipping, and the agent's real production behavior needs ongoing monitoring after it ships, since those catch different kinds of problems. Architect: Offline, I gate every change on running the full golden dataset through both the old and new version side by side, flagging any case that passed before and now fails, using an LLM-as-a-Judge with randomized ordering for scoring to avoid position bias; online, I continue sampling real production traffic against the same classes of metrics after deployment, feeding interesting or failing live cases back into the offline dataset over time. Trade-offs: Offline evaluation is cheap and safe to run on every change but can only catch problems the dataset's authors anticipated; online evaluation catches real unanticipated failures but only after they've already happened to real users. Fail-safe: Relying on offline evaluation alone gives false confidence once production traffic drifts from the dataset's original distribution; the feedback loop from online cases back into the offline set is what keeps that confidence honest over time.
Design a Token Budgeting & Cost-Governance Service: Enforce per-user, per-team, and per-run spend limits in real time before an agent starts an expensive step.
Define: This service has to check and enforce a spend limit before an expensive step starts, not just report on spend after the fact, which means it has to sit in the critical path of every call. Architect: I maintain a real-time counter per user, per team, and per run in a fast, atomic store, checked and decremented before any model or tool call that has a known or estimated cost, rejecting the call outright if any applicable limit would be exceeded, rather than letting it proceed and reconciling the overage later. Trade-offs: Checking a budget synchronously before every costly call adds a small amount of latency to each one, but a budget enforced only by periodic reconciliation can let a burst of calls blow well past the limit before anyone notices. Fail-safe: A race between two concurrent calls both checking the same budget and both passing before either decrements it is the specific risk; the check-and-decrement has to be one atomic operation against the counter, not a separate read followed by a separate write.
Design a Guardrails & Prompt-Injection Firewall: Screen inputs, tool outputs, and final responses with low-latency classifiers without ruining user experience.
Define: This firewall needs to catch malicious or policy-violating content at three points, incoming input, tool results, and the model's own output, while adding minimal perceptible delay to a normal, benign request. Architect: I run a small, fast classifier at each of the three checkpoints rather than a single heavyweight check at the end, tuned so the common, clean case passes through with negligible added latency, and reserve any slower, more thorough check for content the fast classifier flags as suspicious. Trade-offs: A fast classifier at every checkpoint catches more than a single end-of-pipeline check, but a fast classifier is also more prone to false positives that block legitimate content, so its threshold has to be tuned against real traffic, not just tested in isolation. Fail-safe: Content that passes the fast classifier but would have failed the slower, thorough check is the residual risk this design accepts for speed; I route anything the fast check marks as borderline, not just clearly bad, to the slower check rather than only escalating clear failures.
Design a Scoped Permission & Auth Context (ToolContext): Pass user RBAC roles and short-lived OAuth tokens into tools without exposing credentials to the LLM prompt.
Define: Tools need the actual credentials to call real systems, but the model itself must never see a raw credential in its prompt, since anything in the prompt could in principle be reflected back out. Architect: I pass a ToolContext object alongside each tool invocation, populated by the application layer with the requesting user's RBAC roles and a short-lived, narrowly scoped OAuth token, injected directly into the tool's execution environment rather than into the prompt text the model reads; the model only ever sees tool names and non-sensitive parameters, never the context object's contents. Trade-offs: Keeping credentials entirely out of the model's prompt closes off an entire class of credential-leak risk, but it means every tool implementation has to be written to expect a ToolContext parameter rather than reading ambient global credentials, which is more deliberate plumbing. Fail-safe: A long-lived, broadly scoped token sitting in the context would turn any single compromised tool call into a much bigger exposure; tokens are issued short-lived and scoped to exactly what the current task needs, so a leak is bounded in both time and reach.
Design an OpenTelemetry Tracer Decorator for Agents: Automatically wrap agent runs, LLM calls, and tool executions into parent-child trace spans with token and latency attributes.
Define: This decorator needs to add tracing to agent code with minimal intrusion, automatically capturing the parent-child relationship between a run and the calls inside it, without every function having to manually manage span objects. Architect: I implement a decorator that, applied to any agent-run, model-call, or tool-call function, automatically starts a new span as a child of whatever span is currently active in context, attaches standard attributes, latency, token counts where applicable, success or failure, when the function returns or raises, and closes the span, propagating the active-span context automatically so nested decorated calls form the correct parent-child tree with no manual wiring. Trade-offs: A decorator-based approach keeps instrumentation out of the business logic almost entirely, but it only captures what the decorator's standard attribute set anticipates, so anything beyond that, like a custom per-tool metric, still needs to be added manually inside that specific function. Fail-safe: An exception inside a decorated function must still close its span with an error status rather than leaving it open forever; the decorator wraps the call in a try/finally so the span is always closed and tagged with its actual outcome, even when the wrapped function raises.
Level 10: Case studies I, core components
Case studies I, core components — interview questions
Answer in your head first, then open it.
Design a SQL & BI Data Analyst Agent: Translate natural language to SQL across massive data warehouses, self-correct syntax/schema errors, and generate charts safely.
Define: The agent needs to produce correct, safe queries against a real warehouse schema it doesn't have memorized, and recover from its own mistakes rather than failing outright. Architect: It retrieves the actual relevant table and column definitions from a schema catalog before generating SQL, rather than guessing column names, executes the generated query against a read-only replica with a row and cost limit, and on a syntax or schema error, feeds the exact database error message back to the model as a correction signal for a bounded number of retry attempts before giving up and asking the user to clarify. Trade-offs: Running every query against a read-only replica with hard limits protects the production warehouse from an expensive or wrong query, but it means the agent can never answer a question that genuinely requires a write or an unbounded scan. Fail-safe: An unbounded or runaway query is capped by a hard cost and row limit enforced by the warehouse itself, not just by the agent's own intention to limit it, since a generated query's actual cost can't be fully predicted before it runs.
Level 11: Case studies II, products at scale
Case studies II, products at scale — interview questions
Answer in your head first, then open it.
Design an Autonomous Software Engineering Agent (like Devin / SWE-agent): Clone repositories, index codebases, run test suites, edit files, and open GitHub PRs.
Define: This system has to safely explore and change a real codebase, verify its own changes actually work, and hand off a reviewable artifact rather than pushing changes directly. Architect: The agent clones the target repo into an isolated sandbox, builds a searchable index of the codebase to ground its understanding, makes edits inside that same sandbox, runs the existing test suite there to confirm the change actually passes before proceeding, and opens a pull request through the GitHub API rather than merging, so a human reviews the real diff. Trade-offs: Requiring a passing test run before opening the PR catches many broken changes automatically, but it only catches what the existing test suite covers, so a change that passes tests can still be subtly wrong in ways the suite doesn't check. Fail-safe: Never merging directly, only opening a PR, is the structural fail-safe; a confidently wrong fix that passed its own sandbox tests still has to clear human review before reaching the real codebase.
Design a Customer Support & Refund Resolution Agent: Query order databases, enforce policy rules, execute automated refunds under $50, and escalate edge cases with full context.
Define: This needs risk-tiered automation: small, clearly policy-eligible refunds handled automatically, anything larger or ambiguous routed to a human with full context already gathered. Architect: The agent reads order history and the relevant policy through read-only queries, computes eligibility and the dollar threshold deterministically in code, calls a refund tool guarded by an idempotency key only when both checks pass, and routes anything over the threshold or policy-ambiguous to a human queue pre-populated with everything already gathered, rather than making the human start from scratch. Trade-offs: Automating only clearly bounded, low-risk cases keeps the blast radius of a mistake small, but it means a larger share of total refund volume by dollar value still needs a human, capping how much the automation actually saves. Fail-safe: The dollar threshold has to be enforced deterministically in code, never left to the model's own judgment, and every refund call carries an idempotency key so a retried decision can never double-refund.
Design a Deep Research & Financial Report Agent: Run parallel web searches, parse SEC filings, verify numerical citations, and stream a structured report to the user.
Define: Every number in the final report has to trace back to an actual retrieved source, since a hallucinated figure in a financial report is a serious, specific failure mode. Architect: The agent fans out several web searches and filing lookups in parallel since they're independent, retrieves the actual filing and market data through dedicated tools rather than recalling from training knowledge, attaches a citation to every number tracing it to its specific source, runs a verification pass checking every claimed number actually traces to a retrieved source, and streams the report section by section as each part is verified. Trade-offs: Requiring every number to trace to a retrieved source is slower than letting the model freely synthesize figures, but a report with even one hallucinated number undermines trust in the whole document. Fail-safe: A number that can't be traced to a source gets flagged as unverifiable rather than included on the assumption it's probably close enough.
Design an Enterprise Email & Calendar Executive Assistant: Read incoming emails, defend against indirect prompt injection, check calendar availability across time zones, and draft/send replies.
Define: This agent reads untrusted external content (email) and can take real actions (sending, scheduling), so it needs to treat email content as data to read, never as instructions to follow. Architect: Incoming email content is wrapped and explicitly marked as untrusted data in the prompt; any instruction-like text found inside an email is never allowed to directly trigger a tool call, calendar availability is resolved by converting every attendee's stored time zone to a common reference before comparing slots, and any outbound send or calendar write requires a final confirmation step before it fires. Trade-offs: Treating all email content as inherently untrusted closes off most injection paths but means even a legitimate instruction embedded in a real email can't be acted on automatically without the same scrutiny as a malicious one. Fail-safe: An email convincing the agent to send a reply or book a meeting it shouldn't is the specific attack to resist; any action that looks like it originated from email content must be re-confirmed against the actual user's own request before it executes.
Design a Collaborative Browser-Use / Computer-Use Agent: Take screenshots, parse DOM/accessibility trees, execute clicks/typing, and recover when web UIs change.
Define: This agent has to perceive a web page reliably enough to act on it, and recover gracefully when the page's layout changes in a way that breaks its previous assumptions. Architect: It combines a screenshot with the page's accessibility tree, since the tree gives reliable element identifiers that pixel coordinates alone don't, plans an action against a specific identified element rather than a raw coordinate, and after every action re-captures both the screenshot and the tree to confirm the expected state actually changed before planning the next step. Trade-offs: Re-verifying state after every single action adds real latency to each step, but acting blindly on a stale view of the page risks clicking the wrong element entirely once the layout has shifted. Fail-safe: An element that can't be found after a UI change is a common failure; the agent falls back to re-scanning the full page for a semantically equivalent element rather than retrying the exact same failed action.
Design a Medical Triage & Clinical Documentation Agent: Transcribe doctor-patient visits, cross-reference EHR history, mask PII/PHI for HIPAA compliance, and draft clinical notes.
Define: Protected health information has to be handled under strict compliance rules throughout the pipeline, and the agent's output is a draft for a clinician to review, not a final clinical decision. Architect: Audio is transcribed and PHI is detected and masked before any content leaves the compliance boundary to an external model, EHR history is retrieved through an access-controlled lookup scoped to that specific patient and the requesting clinician's permissions, and the draft note is generated from the transcript and history but explicitly marked as a draft requiring clinician sign-off before it becomes part of the record. Trade-offs: Masking PHI before any external processing protects compliance but can degrade the model's ability to reason about identity-linked details, so masked tokens need to be consistently reversible within the trusted boundary once processing is done. Fail-safe: A draft note is never written directly into the patient record; it stays pending until a clinician explicitly reviews and signs it, so an error in the draft can't become part of the official record unreviewed.
Design an Automated Compliance & Contract Review Agent: Ingest 200-page PDFs, extract clauses, compare against legal playbooks, and flag risks with exact page citations.
Define: Every flagged risk has to be traceable to an exact page and clause in the source document, since a legal reviewer needs to verify the agent's claim against the actual contract text. Architect: The document is parsed with page and clause boundaries preserved rather than flattened into undifferentiated text, each extracted clause is compared against the organization's playbook rules, and every flagged risk carries a citation to the specific page and clause it came from, with the comparison logic itself kept as explicit, auditable rules wherever the playbook rule is itself deterministic. Trade-offs: Preserving page and clause structure through parsing is more work than a flat text extraction, but a flagged risk with no exact citation is close to useless to a reviewer who has to re-read the whole document to verify it. Fail-safe: A missed clause that should have been flagged is worse than a false positive here; the comparison pass runs against every extracted clause exhaustively, and any clause the parser couldn't confidently extract is flagged as needing manual review rather than silently skipped.
Level 12: Case studies III, infrastructure and money
Case studies III, infrastructure and money — interview questions
Answer in your head first, then open it.
Design a Self-Healing DevOps / SRE Incident Agent: Ingest PagerDuty alerts, correlate logs/metrics across microservices, diagnose root causes, and execute safe rollback runbooks.
Define: The agent needs to diagnose a live incident quickly while being extremely conservative about what it's allowed to do autonomously, since a wrong automated action during an incident can make things measurably worse. Architect: It ingests the alert and queries logs and metrics across the affected services through read-only tools to form a root-cause hypothesis, correlating timestamps across services to find what changed together, and for remediation it's restricted to executing only pre-approved, tested rollback runbooks, with anything outside that known-safe set requiring human approval first. Trade-offs: Restricting autonomous action to known-safe runbooks lets the agent resolve common, well-understood incidents quickly without waking anyone, but it can't autonomously handle a genuinely novel incident that matches no existing runbook. Fail-safe: Every runbook action is itself idempotent and reversible where possible, and the agent is barred from chaining multiple remediation actions together without a human checking in between, so one bad step can't cascade.
Design an E-Commerce Shopping & Checkout Agent: Search product catalogs, compare specs/prices, manage cart state, and complete payments with idempotency guarantees.
Define: The payment step is the part of this system that can never be allowed to apply twice, so it needs the same idempotency discipline as any financial mutation. Architect: Catalog search and comparison run as ordinary read-only retrieval, cart state is held in a session store scoped to the user, and the final checkout call carries a unique idempotency key generated once per checkout attempt, checked by the payment provider before the charge is applied, so a retried or duplicated checkout request is recognized and not charged again. Trade-offs: Generating and tracking an idempotency key for every checkout adds a small amount of bookkeeping, but without it, any retry after a lost confirmation risks charging the customer twice. Fail-safe: A payment confirmation that's lost in transit must never be interpreted as the payment having failed; the agent checks the actual payment status by the idempotency key before ever re-attempting a charge.
Level 13: Practice, more designs and mock interviews
Practice, more designs and mock interviews — interview questions
Answer in your head first, then open it.
Design a Deterministic Mock Harness for Agent Unit Testing: Record and replay LLM responses and tool outputs (VCR pattern) to run fast, zero-cost CI tests.
Define: This harness needs to let agent logic be tested against realistic model and tool responses without actually calling a real model or a real external API on every test run. Architect: In record mode, the harness intercepts every outbound model and tool call during a real run and saves the call's input and response to a fixture file; in replay mode, used in CI, the harness intercepts the same calls and returns the recorded response for a matching input directly, with no real model or API call made, and fails the test clearly if a call doesn't match any recorded fixture. Trade-offs: Replay-based tests are fast, free, and perfectly repeatable, which is exactly what CI needs, but they only test the agent's own logic against responses captured at one point in time, not whether the real model's actual current behavior still matches what was recorded. Fail-safe: Fixtures that silently go stale as the real model's behavior drifts give false confidence in CI; I periodically re-record fixtures against the real model and treat a large diff between old and newly recorded responses as a signal worth investigating, not just overwriting silently.