CSE Roadmap

Interview questions

Agentic AI Interview Questions

120 agentic AI interview questions with full answers, grouped by level and structured the way a senior candidate should actually answer in an interview: a direct definition, how it works in a real architecture, the trade-offs that show you know when not to use it, and the specific failure mode it causes in production along with how to guard against it.

Jump to a level

Level 3: Prompting and workflows

Prompting and workflows — interview questions

Answer in your head first, then open it.

When should you build a deterministic workflow instead of an autonomous agent?

Define: A deterministic workflow is the right choice whenever the steps needed to complete a task are known in advance and don't depend on information only discovered mid-task. Architect: I write the task as an explicit sequence or graph of steps in code, with the LLM called only for the few steps that genuinely need judgment, such as extracting a field or classifying intent, while everything else stays outside the model's control. Trade-offs: A workflow is cheaper, faster, and far easier to test than an agent, but it can't adapt when a task's shape changes partway through, and it needs a developer to anticipate every branch up front. Fail-safe: The common failure is forcing a workflow onto a task that's secretly open-ended, so it breaks the moment a case outside the anticipated branches shows up; I watch for that symptom and only move to an agent once it actually happens.

What is Tree of Thoughts (ToT) vs. Chain of Thought (CoT) in agent planning?

Define: Chain of Thought has the model produce one linear sequence of reasoning steps toward an answer; Tree of Thoughts has it explore multiple candidate reasoning paths in parallel and evaluate which branch looks most promising before continuing. Architect: CoT is just prompting the model to think step by step along a single thread; ToT generates several possible next steps at each point, scores or self-evaluates them, prunes the weaker branches, and continues expanding only the stronger ones, closer to a search algorithm than a single pass. Trade-offs: ToT finds better solutions on problems with multiple plausible approaches, such as puzzles or planning tasks, but it multiplies the number of model calls needed, since every branch needs its own generation and evaluation. Fail-safe: Without a limit, the branching can grow faster than the budget allows; I cap both the branching factor and the search depth up front so the cost stays bounded regardless of how many promising-looking branches appear.

Level 4: Retrieval and RAG for agents

Retrieval and RAG for agents — interview questions

Answer in your head first, then open it.

How does Agentic RAG differ from traditional naive RAG?

Define: Naive RAG retrieves once, given the user's original query, and generates an answer directly from whatever came back; agentic RAG lets the model itself decide whether to retrieve, what to retrieve, and whether to retrieve again, as part of a reasoning loop. Architect: In agentic RAG, the model can issue its own retrieval queries, inspect what came back, decide the result is insufficient and reformulate the query, or decide a different knowledge source entirely is needed, repeating this until it judges it has enough to answer, rather than following one fixed retrieve-then-generate pass. Trade-offs: Agentic RAG handles multi-hop and underspecified questions far better than a single fixed retrieval pass, but it costs more in latency and token spend, since it can issue several rounds of retrieval before answering. Fail-safe: Without a cap, an agentic retrieval loop can keep retrieving indefinitely on a question it can't actually resolve from the available sources; I cap the number of retrieval rounds and have the agent answer with its best available evidence, clearly flagged as incomplete, rather than loop forever.

What is Corrective RAG (CRAG), and how does the agent grade retrieved documents?

Define: Corrective RAG adds an explicit grading step after retrieval that judges whether the retrieved documents are actually relevant and sufficient before generation proceeds, rather than assuming whatever came back from the retriever is good enough to answer from. Architect: A grading step, often a smaller model, scores each retrieved document as relevant, irrelevant, or ambiguous against the query; if the overall relevance is too low, the system falls back to a different retrieval strategy, such as a web search or a reformulated query, rather than generating from poor evidence anyway. Trade-offs: Grading catches a weak retrieval before it poisons the final answer, improving robustness on queries the index wasn't well suited for, but it adds a model call and latency to every single query, even the ones that would have retrieved well on the first try. Fail-safe: A grading model can itself be wrong, passing a bad set of documents as relevant; I periodically audit the grader's own accuracy against a labeled sample, rather than treating its pass or fail judgment as ground truth once it's in place.

Explain Self-RAG and reflection tokens.

Define: Self-RAG trains or prompts the model to emit special reflection tokens alongside its generation that explicitly judge whether retrieval was needed, whether the retrieved passage actually supports a claim, and whether the generated answer is fully backed by the evidence. Architect: At generation time, the model emits tokens like retrieve or no-retrieve before deciding whether to pull in documents, and tokens marking each generated segment as supported, partially supported, or unsupported by the retrieved evidence, which downstream logic can use to filter out or flag unsupported claims. Trade-offs: Reflection tokens give fine-grained, segment-level confidence about groundedness that a single overall relevance score can't, but training a model to produce them reliably, or getting a general-purpose model to emit them accurately via prompting alone, is harder than it sounds. Fail-safe: A model miscalibrated on its own reflection tokens, confidently marking an unsupported claim as supported, defeats the whole mechanism; I validate the reflection tokens' accuracy against a held-out set of known-grounded and known-ungrounded examples before trusting them in production.

How does an agent decide when to query a vector DB versus answering from internal knowledge?

Define: This decision is about whether the question needs information specific to the organization's own data, requiring retrieval, or is answerable from the model's general training knowledge alone, not requiring it. Architect: I have the agent, or a lightweight classifier ahead of it, check whether the query references anything proprietary, time-sensitive, or specific to the organization's own documents; if so it retrieves, and if the query is clearly general knowledge, it answers directly, rather than retrieving unconditionally on every query regardless of whether it's needed. Trade-offs: Retrieving on every query is simpler to implement and avoids ever answering from potentially outdated internal knowledge, but it adds latency and cost to queries that didn't need it at all; skipping retrieval when it was actually needed risks a confident but wrong answer from stale training knowledge. Fail-safe: The costlier mistake is skipping retrieval when it was needed, since that produces a confidently wrong answer that looks fine; I bias the decision toward retrieving when genuinely uncertain, rather than toward skipping it to save latency.

How do you implement query decomposition and multi-hop retrieval in an agent?

Define: Query decomposition breaks one complex question into several simpler sub-questions that can each be answered by a single retrieval step; multi-hop retrieval then chains those steps together, where a later retrieval depends on what an earlier one found. Architect: The agent first identifies whether a query actually requires multiple pieces of information that aren't in one document, decomposes it into an ordered set of sub-questions, retrieves and answers each in turn, carrying forward what it learned from earlier hops into the query for the next, and finally composes the sub-answers into one response. Trade-offs: Decomposition handles genuinely compound questions a single retrieval pass would fail on, but it requires the agent to correctly recognize that decomposition is needed in the first place, and a bad decomposition at the start propagates its error through every hop that follows. Fail-safe: An early wrong decomposition silently derails the whole multi-hop chain; I have the agent verify each hop's answer against its source before using it as input to the next hop, rather than chaining unverified intermediate answers forward.

How do you combine SQL/structured database querying with unstructured vector search?

Define: Combining the two means routing a query to structured SQL retrieval when it needs precise, filterable facts from a database, and to vector search when it needs semantically similar unstructured text, sometimes using both together for one question. Architect: I have a routing step classify whether a query needs structured filtering (exact dates, amounts, categories), semantic matching over text, or both; for queries needing both, I run the structured filter first to narrow the candidate set, then run vector search only within that narrowed set, rather than searching the full unstructured corpus independently. Trade-offs: Filtering first with SQL before vector search narrows the search space and improves precision, but it only works when the structured filter criteria are actually extractable from the query, which isn't always the case for a loosely phrased request. Fail-safe: A query that should have used the structured filter but got routed to pure vector search instead will return plausible but imprecise results without any obvious error; I validate the router's classification accuracy on a labeled query set, the same way tool-routing accuracy gets validated elsewhere.

What is hybrid search (BM25 + dense embeddings), and why do agents need it?

Define: Hybrid search combines a lexical method like BM25, which matches on exact keywords and terms, with dense embedding similarity, which matches on semantic meaning, merging both sets of results rather than relying on either alone. Architect: Both retrieval methods run against the same query, each returns its own ranked list, and the two lists are merged with a fusion method, such as weighted score combination or reciprocal rank fusion, into one final ranked result set passed to the generation step. Trade-offs: Hybrid search catches both exact-term matches that embeddings sometimes miss, like a specific error code or product SKU, and semantic matches that keyword search alone would miss, but it costs more to run two retrieval systems and needs a fusion step tuned for the specific corpus. Fail-safe: A badly tuned fusion weight can let one method dominate the other and quietly undo the benefit of combining them; I evaluate retrieval quality with both methods' weights varied on a labeled query set rather than picking a fusion weight by intuition alone.

How do you prevent an agent from citing outdated or irrelevant retrieved chunks?

Define: An outdated or irrelevant chunk is a retrieved passage that's no longer accurate, or that's topically similar to the query without actually answering it, either of which can still get cited in a generated answer if nothing checks for it. Architect: I attach freshness metadata, a last-updated date, to every indexed chunk and either filter out or down-rank stale chunks at retrieval time, and run a relevance check, similar to the grading step in corrective RAG, before a chunk is allowed into the context the model generates from. Trade-offs: Filtering on freshness and relevance reduces the chance of citing something wrong, but it requires reliable metadata about when each source was actually current, which isn't always tracked consistently across every ingested document. Fail-safe: A document that's been superseded but still exists unflagged in the index is the classic failure; I treat index freshness as an ongoing operational responsibility rather than a one-time ingestion concern.

How do you handle permissions/ACLs (Role-Based Access Control) during agentic retrieval?

Define: Retrieval-time access control means every retrieved document is filtered against the requesting user's actual permissions before it ever reaches the model's context, so an agent can never surface information the requesting user wasn't allowed to see. Architect: I tag every indexed document with its access-control metadata at ingestion time and apply that same ACL as a hard filter on every retrieval query, scoped to the current user's actual permissions, rather than retrieving broadly and relying on the generation step to politely decline to share anything restricted. Trade-offs: Filtering at retrieval time is the only approach that actually prevents a leak, since anything that makes it into context can in principle be reflected back by the model; relying on the model to withhold restricted information it already saw is not a real security boundary. Fail-safe: A single document ingested without its proper ACL tag becomes retrievable by anyone; I treat missing ACL metadata as a reason to exclude a document from the index entirely by default, rather than defaulting a tagging gap to visible-to-everyone.

When should an agent use web search tools vs. internal knowledge bases?

Define: This decision is about whether the needed information lives in the organization's own curated documents, an internal knowledge base, or in the broader, continuously updated public web, each suited to a different kind of query. Architect: I route queries about the organization's own policies, products, or internal data to the internal knowledge base, and queries about current events, external facts, or anything the internal base structurally can't contain, to web search, sometimes using both when a question spans both domains. Trade-offs: Internal knowledge bases are more trustworthy and controllable but limited in scope to what's been ingested; web search reaches far more current and broader information but introduces source-quality variance and potential prompt injection from untrusted page content. Fail-safe: Content pulled from a web search is untrusted input and must never be treated as instructions; I always treat retrieved web content as data to reason about, never as something that can direct the agent's own behavior.

How do you evaluate retrieval quality separately from agent reasoning quality?

Define: Separating these means measuring whether the right documents were actually retrieved independently from measuring whether the final answer, given those documents, was correct, since a wrong final answer could stem from either stage. Architect: I evaluate retrieval alone using metrics like recall and precision at a given rank against a labeled set of query-to-relevant-document pairs, and evaluate generation alone by feeding a model known-good, pre-selected documents and checking whether it reasons correctly from them, rather than only ever measuring the end-to-end pipeline's final answer. Trade-offs: Separating the two stages makes it possible to tell whether a failing answer was a retrieval problem or a reasoning problem, but it requires maintaining two separate labeled evaluation sets rather than one simpler end-to-end benchmark. Fail-safe: Only ever measuring end-to-end accuracy hides exactly which stage to fix when it drops; I always keep the stage-level metrics running alongside the end-to-end one, specifically so a regression can be localized to retrieval or generation rather than investigated from scratch each time.

How do you handle tables, charts, and multimodal documents in agentic RAG?

Define: Tables, charts, and other non-prose content break standard text-chunking approaches, since naively splitting them into text chunks destroys the structure that gives them meaning. Architect: I parse tables into a structured representation, such as a markdown table or key-value rows, that preserves row-column relationships rather than flattening them into disconnected text, extract charts using a vision-capable model to produce a text description of what the chart actually shows, and index each as its own distinct chunk type rather than merging it into surrounding prose. Trade-offs: Structure-preserving parsing produces far more usable retrieval results for tabular and visual content, but it requires a more sophisticated ingestion pipeline than plain text splitting, and per-document-type parsing adds real engineering and maintenance cost. Fail-safe: A silently mis-parsed table, columns shifted, headers lost, is worse than no table at all, since it produces a confident but wrong-structured answer; I spot-check parsed table output against the original document periodically, rather than trusting the parser's output blindly once it's deployed.

Level 5: Your first agent

Your first agent — interview questions

Answer in your head first, then open it.

What is the difference between a standard LLM workflow (chain) and an autonomous AI agent?

Define: A chain calls the LLM along a fixed sequence of steps the developer wrote in advance; an agent lets the LLM itself decide, at each step, which action to take next based on what it has observed so far. Architect: A chain's control flow lives in code, where step two always runs after step one regardless of what came back; an agent runs a loop where the model reads the current state, picks a tool or action, observes the result, and decides whether to continue or stop. Trade-offs: Chains are predictable, cheap, and easy to test, but they can't handle a task whose steps aren't known in advance; agents handle open-ended tasks but cost more tokens, run longer, and can go down unproductive paths. Fail-safe: The biggest risk with an agent is that it keeps deciding to act when it should stop; I cap the loop with a maximum step count and a clear termination condition so it can't run forever on a task it can't finish.

How does the ReAct (Reason + Act) loop work under the hood?

Define: ReAct interleaves the model's natural-language reasoning with concrete tool calls, one step at a time, so the model explains its thinking before acting and then reads the result before reasoning again. Architect: Each turn, the model emits a thought about what to do next, then an action, a tool call with arguments, the system executes that action and returns an observation, and this thought-action-observation cycle repeats until the model emits a final answer instead of another action. Trade-offs: Writing out the reasoning step improves accuracy on multi-step tasks and makes failures easier to debug from the trace, but it burns extra tokens and adds latency compared to calling tools directly without the reasoning text. Fail-safe: An agent can get stuck reasoning in circles without ever acting; I track whether the last few thoughts are substantially the same and force a stop or a different strategy if they are.

Explain the Plan-and-Execute pattern and how it differs from ReAct.

Define: Plan-and-Execute separates planning from acting: the model first produces a full multi-step plan, and only then does a, possibly different, cheaper executor carry out each step in turn, re-planning only if something goes wrong. Architect: A planner model reasons once, up front, about the whole task and emits an ordered list of steps; an executor then works through that list, calling tools as needed, and a replanning step kicks in only when an execution step fails or returns something unexpected. Trade-offs: Planning up front means fewer expensive reasoning calls to the frontier model and a trace that's easier to audit, but it degrades on tasks where later steps genuinely depend on what earlier steps return, since the plan can't see that information yet. Fail-safe: A stale plan is the main risk; if the world changes mid-execution, the agent may keep executing a plan which no longer makes sense, so I add a validity check before each step and trigger replanning when an assumption the plan depended on turns out false.

What is the Reflection (self-critique) pattern, and how do you implement it?

Define: Reflection has the model critique its own prior output against some criteria and then produce a revised version, rather than accepting the first answer it generates. Architect: After an initial generation, I run a second pass, often with the same model, prompted specifically to find flaws, missing steps, wrong assumptions, unmet requirements, in the first output, then use that critique to produce a corrected version, sometimes repeating this for a fixed number of rounds. Trade-offs: Reflection measurably improves output quality on tasks like code generation and writing, but it at least doubles the token cost and latency of the task, and a model can also talk itself out of a correct answer. Fail-safe: Without a stopping rule, reflection can oscillate between two corrected answers indefinitely; I cap the number of reflection rounds and keep the best-scoring version seen so far rather than always trusting the latest one.

What are the core components of an AI agent (brain, memory, tools, planning)?

Define: An agent is usually described as four cooperating pieces: a brain, the LLM doing the reasoning, memory, what it remembers across and within turns, tools, the actions it can actually take, and a planning mechanism, how it decides what to do next. Architect: The brain receives the current state and available tools, decides an action using one of the planning patterns like ReAct, the tool executes and its result is written back into memory, and the loop continues until a stopping condition is met. Trade-offs: Treating these as separate components makes an agent easier to build, test, and swap pieces of, a different memory backend, a cheaper model, but over-separating them can add coordination overhead for genuinely simple tasks that don't need all four pieces. Fail-safe: The most common break is memory and planning getting out of sync, where the agent acts on stale information; I make sure every tool result is written back into the state the planner reads before the next decision is made.

How do you handle non-determinism in agent outputs?

Define: Non-determinism here means the same input can produce a different agent trajectory or final answer on different runs, since the underlying model's sampling is not perfectly repeatable. Architect: I reduce it where it matters by lowering temperature for decision-making steps, using structured output schemas so the shape of a response doesn't vary even if wording does, and adding deterministic checks, schema validation, business-rule checks, after the model's output rather than trusting the free-text result directly. Trade-offs: Driving temperature to zero makes behavior more repeatable but can also make the agent more repeatably wrong on an edge case, since it loses the variation that sometimes finds a better path. Fail-safe: For anything where the variance itself is risky, such as a refund amount, I don't rely on sampling control alone; I add a deterministic validation layer that rejects or corrects any output outside an allowed range before it's acted on.

Explain the Router pattern in agentic workflows.

Define: A router is a step, often a smaller or cheaper model, that looks at an incoming request and decides which specialized downstream path, agent, or tool should actually handle it. Architect: The router receives the raw request, classifies it against a known set of categories or destinations, and forwards it, with whatever context is needed, to the matching handler, rather than one single agent trying to handle every type of request itself. Trade-offs: Routing keeps each downstream handler simple and specialized, and lets you use a cheap model for the classification step, but a misrouted request fails silently in the wrong handler unless that handler can detect it doesn't actually fit. Fail-safe: I give each downstream handler a way to reject a request that doesn't match its expected shape and bounce it back to the router or to a human, rather than letting it attempt a task it was never built for.

How do you handle ambiguous user intent before an agent starts executing tools?

Define: Ambiguous intent means the request could reasonably mean more than one thing, so acting on the first interpretation risks doing the wrong thing, possibly irreversibly. Architect: Before any tool with a real-world side effect runs, I have the agent check whether its interpretation is confident enough to proceed; if not, it asks a clarifying question instead of guessing, and only read-only, reversible actions are allowed to run during that uncertain phase. Trade-offs: Asking for clarification adds a round trip and can frustrate a user who expected a one-shot answer, but skipping it risks a wrong, possibly irreversible action being taken on a guess. Fail-safe: The specific risk is a confidently-wrong guess on a mutating action; I gate any write or spend action behind either a high-confidence intent match or an explicit confirmation step, so ambiguity defaults to asking, not acting.

Level 6: Tools, MCP and computer use

Tools, MCP and computer use — interview questions

Answer in your head first, then open it.

How does function calling / tool use actually work at the LLM API level?

Define: Function calling lets a developer describe a set of available functions, name, description, parameter schema, to the model; the model, instead of only returning text, can return a structured request to call one of those functions with specific arguments. Architect: The API call includes the tool schemas alongside the prompt; the model's response either contains normal text or a structured tool-call object naming the function and its arguments; the application code actually executes that function and sends the result back to the model in a follow-up message so it can continue. Trade-offs: This keeps the actual execution, and any side effects, entirely in the application's control rather than the model's, which is safer, but it also means every real action requires an extra round trip to the model to decide and another to continue after the result. Fail-safe: The model can call a function with plausible-looking but invalid arguments; I validate every tool call's arguments against the schema before execution and return a clear error back to the model rather than letting a malformed call hit a real system.

What is the Model Context Protocol (MCP), and what problem does it solve?

Define: MCP is a standard protocol for connecting an LLM application to external tools and data sources, so a tool provider can expose its capabilities once in a standard shape rather than every application integrating with every tool in a custom, one-off way. Architect: An MCP server exposes a set of tools, resources, and prompts over a standard interface; any MCP-compatible client, the agent's host application, can discover and call those tools without needing bespoke integration code for each one. Trade-offs: Standardizing on MCP means new tools can be plugged in with far less integration work, but it adds a protocol layer and a server process to operate and secure, compared to a tightly coupled, custom integration built for one specific tool. Fail-safe: An MCP server is effectively a new attack surface, since it can expose powerful tools to any client that connects; I scope each server's tools tightly and never expose a write-capable tool without its own permission check independent of the protocol layer.

How do you structure tool descriptions and schemas so the LLM picks the right tool reliably?

Define: A tool description is the model's only information about what a tool does and when to use it, so its clarity directly determines how often the model picks the right tool. Architect: I write a description that states exactly what the tool does, when to use it versus a similar tool, and what each parameter means, use a strict typed schema rather than a free-text parameter, and give concrete examples in the description when two tools could plausibly be confused. Trade-offs: Very detailed descriptions improve tool-selection accuracy but consume context window budget for every single call, which matters more as the number of available tools grows. Fail-safe: Two similarly named or similarly described tools are the classic failure; I explicitly call out the distinction between any pair of tools that could be confused, directly in both of their descriptions, rather than assuming the model will infer it.

How many tools can a single agent handle before tool-selection accuracy degrades?

Define: Tool-selection accuracy measurably degrades well before a model runs out of context space for the tool list, typically somewhere in the range of ten to twenty tools depending on how distinct they are, because the model has to discriminate between more and more similar-looking options. Architect: I measure this empirically for the actual tool set rather than assuming a number, running an evaluation set of tasks against the full tool list and checking how often the right tool was chosen as the count grows. Trade-offs: Keeping a small tool list per agent call is more reliable, but splitting tools across multiple specialized agents or retrieval steps adds routing complexity and another place a request can be misdirected. Fail-safe: Once the measured accuracy for a given tool list drops below an acceptable threshold, I don't try to fix it with a better prompt alone; I split the tools into groups and introduce dynamic filtering instead.

How do you dynamically load or filter tools when you have 100+ APIs (RAG for tools)?

Define: Tool RAG applies retrieval to the tool list itself: instead of giving the model every available tool every time, a retrieval step first selects only the handful of tools likely relevant to the current request. Architect: Each tool's description is embedded and indexed like a document; when a request comes in, the system retrieves the top-k most relevant tool descriptions by similarity, or by a lighter classifier, and includes only those in that call's prompt, rather than all hundred-plus. Trade-offs: This keeps each individual call's tool list small and accurate, but it adds a retrieval step that can miss the genuinely right tool if its description doesn't match the query's wording well. Fail-safe: A tool that's never retrieved is effectively invisible to the agent even though it exists; I track retrieval recall on a labeled set of tasks against their expected tools, so a tool silently falling out of consideration gets caught rather than discovered in production.

How should a tool format error messages back to the LLM so it can self-correct?

Define: A tool error message should tell the model specifically what went wrong and what a valid call would look like, not just that something failed. Architect: I return structured errors with a clear reason code, a human-readable explanation of what was wrong with the specific call made, and, where possible, a suggestion of the correct parameter or next step, rather than a bare exception message or an empty response. Trade-offs: Rich error messages cost a little more in response size and in writing, but a vague error is exactly what causes a model to retry the identical failing call over and over, which costs far more. Fail-safe: An error message a model can't act on is as bad as no error at all; I specifically test that each error type leads the model to a different, corrected call rather than a repeat of the same one, during evaluation.

What is the difference between parallel tool calling and sequential tool calling?

Define: Sequential tool calling runs one tool call, waits for its result, and only then decides the next call; parallel tool calling lets the model request several independent tool calls at once, executed concurrently. Architect: For calls with no dependency between them, such as fetching weather for three different cities, the model returns multiple tool-call requests in a single turn, the application executes them concurrently, and all results come back together before the model reasons about them. Trade-offs: Parallel calls cut total latency substantially when calls are independent, but they only work when the calls really don't depend on each other's results; forcing dependent calls into parallel execution produces wrong or stale results. Fail-safe: The risk is the model requesting parallel calls that actually do depend on each other; I validate that tools marked as parallel-safe have no shared state or ordering requirement, and keep genuinely dependent calls sequential even if that costs latency.

How do you handle long-running tools (e.g., a job that takes 5 minutes to finish)?

Define: A long-running tool can't simply block the agent's turn for five minutes, since that would tie up the whole interaction; it needs to be handled asynchronously instead. Architect: The tool call kicks off the job and immediately returns a job ID or ticket rather than the final result; the agent then either polls a status tool periodically or the system uses a callback or webhook to resume the agent's turn once the job actually completes. Trade-offs: Asynchronous handling keeps the agent responsive and able to work on other things while waiting, but it adds real complexity: the agent's state has to be checkpointed and safely resumed later, exactly once, not zero or twice. Fail-safe: A poll loop that never terminates is a real risk if the job silently fails; I give every long-running job a maximum wait time and a terminal failure state the agent can detect, not just a success state.

How do you enforce strict JSON schema adherence (Structured Outputs / Constrained Decoding)?

Define: Constrained decoding restricts the model's token generation at each step to only tokens that keep the output conforming to a target schema, guaranteeing the final output is valid JSON matching that schema rather than hoping the model happens to produce it. Architect: I pass the target JSON schema to the API's structured-output feature, which constrains sampling so invalid tokens that would break the schema are excluded at each generation step, rather than generating freely and validating afterward. Trade-offs: This gives a strong guarantee on shape, which simplifies downstream parsing enormously, but it constrains only the structure, not the correctness of the values inside it, and very complex or deeply nested schemas can degrade output quality. Fail-safe: A valid-shaped but semantically wrong output can still slip through; I pair schema-level constraints with a separate semantic validation pass on the field values themselves, since shape conformance and correctness are two different guarantees.

What is code execution (CodeAct) as a tool, and why is it often better than JSON tool calls?

Define: CodeAct has the model write and execute actual code, typically Python, as its action, rather than emitting a single JSON tool call; the code itself can call multiple tools, loop, and branch within one action. Architect: The model generates a code snippet that calls the available tool functions directly as normal function calls, the snippet runs in a sandbox, and its output or return value becomes the observation fed back to the model, replacing several separate JSON tool-call round trips with one. Trade-offs: Code execution lets the model express loops, conditionals, and multi-tool compositions in a single action, cutting the number of model round trips substantially, but it requires a genuinely secure sandbox, since arbitrary code execution is a much larger attack surface than a constrained JSON call. Fail-safe: Letting a model write and run arbitrary code demands strict sandboxing; I run it in an isolated environment with no network access beyond an explicit allowlist and a hard resource and time limit, so a runaway or malicious snippet can't escape or exhaust the host.

How do you handle API rate limits and retries inside agent tools?

Define: A tool call can fail specifically because an underlying API's rate limit was hit, which is a different failure mode than a genuine error and should be handled differently. Architect: I implement exponential backoff with jitter for rate-limit-specific error codes, cap the total number of retries, and surface a clear temporarily-unavailable state to the agent rather than a generic failure, so the agent doesn't mistake a rate limit for a permanent error. Trade-offs: Retrying with backoff recovers gracefully from transient rate limiting, but it adds latency to the affected request, and too aggressive a retry policy can itself worsen the rate-limiting problem for other callers sharing the same API key. Fail-safe: An agent that doesn't distinguish rate limiting from real failure can give up on a task it should have just retried, or worse, retry a non-idempotent call that actually succeeded; I pair retry logic with strict idempotency on every mutating call.

What is idempotency, and why is it critical for agent tools that modify data?

Define: An idempotent operation produces the same end result no matter how many times it's executed with the same input, which matters because an agent may retry a call after an ambiguous or lost response without knowing whether the original attempt actually succeeded. Architect: I give every mutating tool call a unique idempotency key generated once per logical action; the backend checks that key before applying the change, so a retried call with the same key is recognized and skipped rather than applied a second time. Trade-offs: Adding idempotency keys requires backend support and a bit of extra bookkeeping on every mutating call, but without it, any retry of a write operation risks double-charging, double-sending, or otherwise duplicating a real-world effect. Fail-safe: This is exactly the mechanism that prevents a retried billing call from actually charging twice even if the agent does retry several times; the idempotency key guarantees the underlying charge only ever applies once regardless of how many retries happen.

How do you prevent an agent from hallucinating tool parameters?

Define: Parameter hallucination is the model inventing a plausible-looking but fabricated value for a tool argument, such as guessing an account ID instead of looking one up. Architect: I require the model to retrieve or confirm any identifier-like parameter through a lookup tool before using it in a mutating call, rather than allowing it to be typed freely, and I validate every argument against known-valid values or formats before execution. Trade-offs: Forcing a lookup step before use adds a round trip for every such parameter, but accepting an unverified, model-generated identifier risks acting on a value that was never real. Fail-safe: Validation at execution time is the actual backstop; even with a lookup-first design, I reject any mutating call whose identifier doesn't match a value actually returned by a prior lookup in that same session, rather than trusting it just because it looks well-formed.

How do you design read-only tools versus write/mutating tools?

Define: A read-only tool can be called freely since it has no side effects; a mutating tool changes real state, so it needs stricter controls around when and how it's allowed to run. Architect: I mark every tool explicitly as read-only or mutating in its definition, let the agent call read-only tools without restriction, and gate mutating tools behind extra checks, confirmation, permission scoping, idempotency keys, and sometimes human approval, that read-only tools don't need. Trade-offs: This separation adds some design overhead up front, since every tool has to be classified correctly, but it lets the system apply much lighter restrictions to the common, safe case of reading without slowing it down to match the strict case of writing. Fail-safe: A tool mislabeled as read-only when it actually has a side effect is a real risk; I treat any tool that touches an external system with write access as mutating by default unless proven otherwise, rather than trusting its name or description.

How do you version tools in production without breaking active agent sessions?

Define: Versioning a tool means changing its schema or behavior while sessions that started under the old version are still in flight and may still expect the old shape. Architect: I keep the old tool version available and callable under its own name or version tag while introducing the new version separately, route new sessions to the new version, and only retire the old one once no active session still depends on it. Trade-offs: Running two versions in parallel for a transition period costs some maintenance overhead, but changing a tool's schema out from under an in-flight session risks a call that was valid when planned becoming invalid by the time it executes. Fail-safe: An in-flight session calling a now-removed tool version fails outright; I track which sessions are still active against which tool version before retiring anything, rather than retiring on a fixed calendar schedule regardless of what's still running.

Level 7: Memory and context engineering

Memory and context engineering — interview questions

Answer in your head first, then open it.

What is the difference between short-term memory and long-term memory in agents?

Define: Short-term memory is what the agent holds within its current context window for the current task or conversation; long-term memory is information persisted outside that window and retrieved back in when relevant, surviving across sessions. Architect: Short-term memory is just the running conversation and scratchpad state inside the prompt; long-term memory lives in an external store, such as a vector database or a structured record, and a retrieval step pulls the relevant pieces back into the context window when needed. Trade-offs: Long-term memory lets an agent remember facts across sessions without an ever-growing context window, but retrieval can miss or misretrieve relevant memories, where short-term memory, while limited in size, is always fully present and accurate for what it holds. Fail-safe: Treating something as remembered just because it was said once in long-term storage is risky if retrieval fails silently; I verify retrieval actually surfaced the expected memory in evaluation rather than assuming storage alone guarantees recall.

Explain the four types of agent memory: working, episodic, semantic, and procedural.

Define: Working memory is the current task's active context; episodic memory records specific past events or interactions; semantic memory holds general facts and knowledge; procedural memory holds learned skills or routines for how to do something. Architect: Working memory lives in the active prompt; episodic memory is typically stored as timestamped, retrievable records of past interactions; semantic memory is stored as facts, often in a knowledge base or vector store; procedural memory is often encoded as reusable prompts, scripts, or fine-tuned behavior rather than retrieved facts. Trade-offs: Separating memory this way clarifies what storage and retrieval mechanism fits each kind, but implementing all four adds real architectural complexity, so many production agents only implement working and episodic memory and treat the rest as out of scope. Fail-safe: Conflating episodic and semantic memory is a common mistake, where a one-off event gets treated as a general fact; I keep them in separate stores with separate retrieval paths so a single past incident doesn't get generalized into a wrong standing fact.

How do you manage context window bloat during long multi-step agent trajectories?

Define: Context bloat happens when every tool call, observation, and reasoning step accumulates in the prompt across a long trajectory until the window fills up or relevant signal gets buried. Architect: I summarize or drop older steps that are no longer directly relevant to the current decision, keep only the most recent raw tool outputs in full while compressing older ones into short summaries, and store the full trajectory externally for audit even as the live context stays trimmed. Trade-offs: Trimming keeps the agent responsive and within budget, but overly aggressive trimming can discard information the agent actually needs later, causing it to repeat work or ask a question it already had the answer to. Fail-safe: The specific risk is trimming something load-bearing; I keep a small, explicitly protected set of facts, the original goal, key constraints, that's never subject to trimming, separate from the general rolling trajectory that is.

When do you use conversation summarization vs. sliding window truncation?

Define: Summarization compresses older turns into a shorter representation that preserves their meaning; sliding window truncation simply drops the oldest turns outright once a size limit is reached. Architect: I use truncation when older turns are genuinely no longer relevant to the task at hand, and summarization when they still carry information needed later but can't all fit verbatim, running a periodic summarization pass that folds old turns into a running summary kept alongside recent raw turns. Trade-offs: Summarization preserves more information per token than truncation, but it costs an extra model call to generate and risks losing or distorting a specific detail in the compression; truncation is free but loses whatever falls off the edge completely. Fail-safe: A summary that quietly drops a specific number or constraint is dangerous because it looks complete; I keep any safety-critical fact, like a hard budget limit, outside the summarized portion entirely, in a protected slot that summarization never touches.

How do you implement persistent state across multiple user sessions (checkpointing)?

Define: Checkpointing saves an agent's state at defined points so a session can be paused and later resumed, or recovered after a crash, without starting the task over. Architect: I serialize the agent's state, the conversation history, the task progress, and any in-flight tool calls at each significant step, write that checkpoint to durable storage keyed by session ID, and on resume, load the latest checkpoint back in rather than replaying from the start. Trade-offs: Frequent checkpointing gives fine-grained recovery and lets sessions be paused and resumed cheaply, but it adds write overhead and storage cost on every step, so checkpointing too often becomes its own performance problem. Fail-safe: Resuming from a stale or corrupted checkpoint can replay a step that already partially executed; I pair checkpointing with idempotent tool calls so resuming from any checkpoint, even one slightly behind reality, doesn't double-apply a side effect.

What is the "lost in the middle" phenomenon, and how does it affect agent memory?

Define: Lost in the middle is the observed tendency for a model to pay less attention to information placed in the middle of a long context window compared to information near the beginning or end. Architect: This matters directly for agent memory because a fact retrieved into the middle of a long trajectory, surrounded by many other tool outputs, is more likely to be effectively ignored than the same fact placed near the start, as a system instruction, or the end, as the most recent observation. Trade-offs: Restructuring context to put critical information at the edges improves attention to it, but it constrains how context gets organized and can conflict with simply appending information in the order it naturally arrives. Fail-safe: I don't assume a fact in context is actually being used; for anything safety-critical, I re-inject it near the end of the prompt just before the decision step, rather than trusting its position from earlier in a long trajectory.

How do you handle memory pruning and forgetting outdated facts?

Define: Memory pruning removes or deprioritizes stored facts that are no longer accurate or relevant, which matters because long-term memory that only ever accumulates eventually contains contradictions and stale information. Architect: I attach a timestamp and, where possible, a confidence or validity window to each stored memory, run periodic or on-access checks that flag memories contradicted by newer information, and either archive, rather than hard-delete, outdated entries or mark them as superseded so retrieval prefers the newer fact. Trade-offs: Active pruning keeps retrieval results fresher and reduces the chance of acting on stale data, but it adds ongoing maintenance cost and risks pruning something that was actually still valid. Fail-safe: Hard-deleting a memory that turns out to still matter is unrecoverable; I archive rather than delete, so a pruning mistake can be reversed, and I prefer mark-as-superseded over removal wherever the application allows it.

How do you store and retrieve user preferences over time (e.g., Mem0, Zep)?

Define: Preference memory is a specific kind of long-term memory that tracks a user's stated or inferred preferences, communication style, past choices, so future interactions don't have to be re-asked or re-derived each time. Architect: I extract explicit or inferred preferences from conversations as structured facts, store them keyed by user ID in a dedicated preference store rather than mixed into general episodic memory, and retrieve the relevant subset at the start of a new session rather than searching for them reactively. Trade-offs: A dedicated preference store makes retrieval fast and predictable since it's structured rather than searched, but it requires an explicit extraction step to decide what counts as a preference worth storing, which is itself a judgment call that can get it wrong. Fail-safe: A wrongly inferred preference, once stored, can quietly steer every future interaction in the wrong direction; I let users see and correct their stored preferences directly, rather than only inferring and never surfacing them.

What is a scratchpad or state object in graph-based agents?

Define: A scratchpad, or state object, is the shared, structured data that's passed between nodes in a graph-based agent workflow, carrying whatever the current step needs to know, as distinct from the free-text conversation history. Architect: Each node in the graph reads fields it needs from the state object, does its work, and writes its results back into specific fields, with the graph's edges determining which node runs next based on the state's current values, rather than everything being crammed into one conversational thread. Trade-offs: A typed state object makes data flow between steps explicit and far easier to debug than inferring what a free-text history means at a given point, but designing the right schema up front requires knowing in advance what each step will actually need. Fail-safe: A node reading a field that hasn't been set yet, because an earlier node was skipped, is a common bug; I give state fields explicit defaults and validate required fields are present before a node runs, rather than assuming upstream nodes always populated them.

How do you isolate memory per tenant/user in a multi-tenant SaaS application?

Define: Memory isolation means one tenant's or user's stored memories, preferences, and history are never visible to or retrievable by another tenant's agent session, even though they may share the same underlying infrastructure. Architect: Every memory record is tagged with a tenant or user ID at write time, every retrieval query is scoped by that same ID as a hard filter, not just a ranking signal, and I use separate storage namespaces or row-level security rather than relying on application code alone to remember to filter correctly. Trade-offs: Enforcing isolation at the storage layer, not just in application logic, is more work up front but closes off an entire class of cross-tenant data leaks that a forgotten filter in one code path could otherwise cause. Fail-safe: A missing tenant filter in just one retrieval path is enough to leak data across tenants; I treat this as a security boundary, not a convenience feature, and test cross-tenant isolation explicitly rather than trusting that application code always remembers to filter.

When would you use a Knowledge Graph (GraphRAG) over a vector database for agent memory?

Define: GraphRAG represents memory as explicit entities and the relationships between them, which suits questions about how things connect; a vector database represents memory as embeddings ranked by similarity, which suits questions about what's semantically close to a query. Architect: I use a knowledge graph when the task needs multi-hop reasoning over explicit relationships, such as who reports to the person who approved this, and a vector database when the task is closer to finding the passage most relevant to a question, since similarity search doesn't naturally traverse relationships. Trade-offs: A knowledge graph answers relationship questions a vector search structurally cannot, but it requires an explicit extraction and modeling step to build the graph, where embeddings can be generated automatically from raw text with far less upfront structure. Fail-safe: Forcing a relationship question through plain vector search produces a plausible-looking but structurally wrong answer rather than an obvious failure; I route queries that are clearly asking about connections between entities to the graph rather than similarity search.

How do you handle conflicting facts retrieved from long-term memory?

Define: A conflict arises when two retrieved memories about the same subject disagree, often because one is outdated and the pruning or update process hasn't caught up with the other. Architect: I prefer the more recently timestamped memory by default, surface both to the agent with their timestamps when the conflict is material rather than silently picking one, and let the agent, or a human, resolve genuinely ambiguous conflicts rather than hard-coding a single resolution rule for every case. Trade-offs: Always trusting the newest fact works well for things that genuinely change over time, but it's wrong for facts that don't change, where an older, correct memory could be overridden by a newer, mistaken one. Fail-safe: Silently picking one of two conflicting facts hides a real data-quality problem; I log every detected conflict so the underlying memory store's integrity issue gets fixed, not just papered over at retrieval time.

What is context caching (prompt caching), and how does it reduce agent costs?

Define: Prompt caching lets a provider reuse the computation already done for a prefix of the prompt that was seen in an earlier call, charging less, and responding faster, for the cached portion than for newly processed tokens. Architect: I structure prompts so the stable part, system instructions, tool definitions, long reference documents, comes first and the variable part, the specific user turn, comes last, since caching works on a matching prefix, maximizing how much of each call can hit the cache. Trade-offs: Caching cuts cost and latency substantially for agents making many calls with a large, mostly-stable prompt prefix, but any small change to that stable prefix invalidates the cache for every call that follows, so prefix stability has to be a deliberate design choice. Fail-safe: An accidental change early in the prompt, like a timestamp placed too close to the front, silently kills the cache hit rate without any error; I keep genuinely variable content, including timestamps, as late in the prompt as possible and monitor cache hit rate directly.

How do you pass large payloads (like a 50-page PDF or CSV) between tools without blowing up the prompt?

Define: A large payload passed directly through the prompt would consume most or all of the context budget on one artifact; instead, it should be passed by reference, not by value. Architect: I write the large payload to a file or object store and pass only a reference, such as a file ID or path, between tools; a tool that actually needs the content reads it directly from storage, or a chunked or summarized view of it, rather than the model re-emitting the whole thing through the context window. Trade-offs: Passing by reference keeps the context window free for reasoning, but it means any tool needing the content must have its own access path to that storage, adding a bit of plumbing that passing the raw content directly through the prompt would have avoided. Fail-safe: A tool that silently falls back to reading the full raw payload into context when its summarized or chunked path fails can blow the budget without warning; I make that fallback fail loudly instead, forcing the chunked path to actually work rather than degrading silently into the expensive one.

Level 8: Multi-agent systems and frameworks

Multi-agent systems and frameworks — interview questions

Answer in your head first, then open it.

How do you decide between single-agent and multi-agent architectures?

Define: The decision comes down to whether one agent with enough tools and context can handle the task coherently, or whether the task genuinely needs distinct roles, perspectives, or parallel work that one agent's single context can't hold well. Architect: I start with a single agent for any linear workflow and move to multiple only when a single prompt becomes overloaded with too many tools or personas, or when the task benefits from genuinely distinct roles, like separating a researcher that gathers context from a reviewer that validates it. Trade-offs: Multi-agent setups improve modularity, context isolation, and parallelism, but they multiply token costs, add real latency from coordination, and make debugging significantly harder since a failure could originate in any of several agents. Fail-safe: The most common failure mode is treating multi-agent as a default rather than a response to an actual overload symptom; I only split into multiple agents once a single-agent version has actually been tried and shown a specific, concrete limitation.

Compare Supervisor (Hierarchical), Swarm (Handoff), and Sequential multi-agent topologies.

Define: Supervisor topology has one orchestrating agent that delegates to and coordinates several worker agents; swarm, or handoff, topology lets any agent hand control directly to another peer agent without a central coordinator; sequential topology runs a fixed chain of agents, each handing to the next in a set order. Architect: A supervisor routes tasks to the right worker and synthesizes their results back into one answer; a swarm lets an agent that recognizes a request outside its scope transfer the conversation directly to a more suitable peer; a sequential pipeline just runs agent A, then B, then C, in that fixed order every time. Trade-offs: Supervisor topology centralizes control and is easier to reason about but makes the supervisor a bottleneck and single point of failure; swarm is more flexible for unpredictable routing but harder to trace; sequential is simplest and most predictable but can't adapt its order to what a task actually needs. Fail-safe: The failure that cuts across all three is losing track of who's currently responsible for the task; I always maintain an explicit current-owner field in shared state so a dropped handoff or crashed supervisor is detectable rather than silently stalling the whole system.

How do you pass state between two specialized agents cleanly?

Define: Clean state-passing means the receiving agent gets exactly the structured information it needs to continue the task, not the entire raw conversation history of the agent that came before it. Architect: I define an explicit schema for what gets handed off, the task, relevant facts gathered so far, constraints, rather than passing the full transcript, so the receiving agent starts with a clean, scoped brief instead of having to re-derive what matters from a long history. Trade-offs: A structured handoff schema is more deliberate to design but keeps each agent's context focused and cheap; passing the full raw history is lazier to implement but wastes tokens and risks the receiving agent fixating on irrelevant earlier details. Fail-safe: Losing a critical fact in the handoff because it wasn't in the schema is the real risk; I treat the handoff schema as something to revise whenever a downstream agent is found to be missing information it needed, rather than fixing it once and assuming it's complete forever.

How do you prevent infinite "ping-pong" loops between two debating agents?

Define: Ping-pong is when two agents keep handing a task back and forth to each other without converging, often because each is waiting for the other to concede or clarify something neither is actually resolving. Architect: I cap the number of back-and-forth turns allowed between any two agents on a given exchange, and if the cap is hit without resolution, escalate to a third party, a judge agent or a human, rather than letting the two continue indefinitely. Trade-offs: A hard turn cap guarantees termination, but it can cut off a genuinely productive exchange that just needed one more round; setting the cap too low trades away real quality for safety. Fail-safe: Detecting the ping-pong before the cap is hit is better than just capping it; I check whether consecutive turns are substantially repeating the same positions and trigger early escalation if so, rather than waiting for the hard limit every time.

What is the "telephone game" problem in multi-agent systems (information loss across handoffs)?

Define: The telephone game problem is information quietly degrading or getting distorted each time it passes through another agent's summarization or interpretation on the way to its final destination, the same way a message changes passing person to person. Architect: I minimize the number of summarization or rephrasing steps a piece of critical information passes through before it's actually used, and where a precise fact, a number, an ID, must survive a handoff, I pass it as a structured field rather than embedded in prose that gets re-summarized. Trade-offs: Passing structured fields alongside prose protects specific facts from drift, but it requires identifying in advance which facts are precision-critical, which isn't always obvious until a failure has already shown it. Fail-safe: Once a telephone-game distortion is found, I don't just fix that one instance; I check whether the same fact passes through other handoff chains in the system and protect it structurally everywhere it does, not only where it was caught.

How do you handle shared memory vs. private memory across multiple agents?

Define: Shared memory is state every agent in the system can read and write; private memory belongs to one agent alone and isn't visible to the others, which matters because not every agent should see everything every other agent knows. Architect: I put only what genuinely needs to be coordinated on, the task's overall goal, the current owner, results that downstream agents depend on, into shared state, and keep each agent's own working notes, intermediate reasoning, and scratch calculations private to it. Trade-offs: More shared memory makes coordination easier since agents don't need explicit handoffs for everything, but it also means more opportunity for one agent's mistake or hallucination to directly corrupt what every other agent sees. Fail-safe: An agent writing an unreviewed, wrong value directly into shared state can propagate that error to every other agent instantly; I validate what gets written to shared state at the point of writing, not just at the point of reading.

When should agents communicate via natural language vs. structured state schemas?

Define: Natural language communication passes free-text messages between agents; structured schemas pass typed, validated fields, which matters because the choice determines how reliably the receiving agent can act on what it gets. Architect: I use structured schemas for anything a downstream agent needs to act on precisely, an amount, a decision, a status, and natural language only for genuinely open-ended content, like a researcher's narrative findings that a reviewer agent needs to read and judge holistically rather than parse. Trade-offs: Structured schemas are far more reliable for anything precise but require upfront agreement on the schema's shape; natural language is more flexible and needs no upfront schema design, but it reintroduces the ambiguity and parsing risk structured data was meant to avoid. Fail-safe: An agent trying to extract a precise number out of a natural-language message is a common silent failure point; wherever a downstream agent needs to act on a specific value, I require it to come through a structured field, never parsed out of prose after the fact.

How do you resolve conflicts when two sub-agents return contradictory findings?

Define: A conflict here is two sub-agents independently investigating the same question and coming back with incompatible answers, which can't both be acted on as given. Architect: I have a dedicated resolution step, often a separate judge or supervisor agent, compare both findings against the original evidence each sub-agent used, rather than simply picking one arbitrarily or averaging incompatible claims, and escalate to a human when the judge itself can't confidently resolve it. Trade-offs: A dedicated judge step adds latency and cost to every task where sub-agents could disagree, but silently picking one contradictory finding over the other risks acting on the wrong one without ever surfacing that a conflict existed. Fail-safe: An unresolved or wrongly resolved conflict is dangerous specifically because it looks like a confident, single answer afterward; I log every detected contradiction even after resolution, so a pattern of frequent disagreement between two sub-agents gets investigated rather than quietly resolved over and over.

What is the Agent-to-Agent (A2A) protocol, and how does it complement MCP?

Define: A2A is a protocol for agents, potentially built by different teams or vendors, to discover each other's capabilities and communicate directly as peers; MCP, by contrast, standardizes how one agent connects to tools and data sources, not to other agents. Architect: An agent exposes an A2A-compatible interface describing what tasks it can perform; another agent discovers and calls it the same way it would call any peer agent, while that same agent might separately use MCP to reach its own tools and data underneath. Trade-offs: Together, the two protocols let an ecosystem of independently built agents and tools interoperate without custom integration for every pairing, but adopting both means tracking and securing two separate protocol surfaces rather than one unified integration layer. Fail-safe: An externally reachable A2A endpoint is a new trust boundary, since it's effectively letting another organization's agent act within your system; I treat every inbound A2A call with the same scrutiny as an external API call, not as an internal, implicitly trusted request.

How do you scale multi-agent systems horizontally under high concurrent traffic?

Define: Horizontal scaling means handling more concurrent agent sessions by running more instances of the same agent processes across more machines, rather than making any single instance faster. Architect: I keep agent instances stateless where possible, with session state externalized to shared storage, the checkpointing approach covered under memory, so any instance can pick up any session, and put a load balancer or queue in front to distribute incoming sessions across the available instances. Trade-offs: Externalizing state to support horizontal scaling adds latency to every state read and write compared to keeping it in the agent process's own memory, but it's what makes adding more instances under load actually possible. Fail-safe: Two instances both believing they own the same session is a dangerous race condition; I use an explicit lock or lease on session ownership so only one instance is ever actively processing a given session's next step at a time.

How do you design a "Critic" or "Judge" agent to validate another agent's work?

Define: A critic or judge agent independently evaluates another agent's output against explicit criteria, acting as a check rather than a collaborator producing the original work. Architect: I give the judge a separate, clearly defined rubric, correctness, completeness, policy compliance, distinct from whatever instructions produced the original work, have it evaluate the output against the original request and evidence rather than against the producing agent's own reasoning, and keep it from simply re-deriving the same answer the same way. Trade-offs: A dedicated judge catches a class of errors the original agent is structurally unlikely to catch in itself, but it doubles the cost of every task it reviews and can introduce its own biases, such as favoring longer or more confident-sounding answers. Fail-safe: A judge that shares the same blind spot or bias as the agent it's judging provides false confidence; I periodically validate the judge itself against a human-labeled sample, not just trust that having a second model constitutes a real check.

How do you debug a failure when five different agents touched the request?

Define: Debugging a multi-agent failure means identifying which agent, and which specific step within that agent, introduced the problem, out of several agents that each touched the request along the way. Architect: I require every agent to emit a structured trace of its inputs, decisions, and outputs tagged with a shared request ID, so the full path through all five agents can be reconstructed and replayed in order, rather than trying to infer what happened from the final output alone. Trade-offs: Full per-agent tracing adds logging overhead and storage cost on every request, but without it, a failure that only shows up in agent five might actually have originated in agent two, and there'd be no way to tell. Fail-safe: A trace with a gap at exactly the agent that failed is the worst case; I make trace-emission mandatory and treat a missing trace segment as its own alert, not just missing debug information, since it usually means that agent crashed before it could log anything.

What happens to user latency in multi-agent architectures, and how do you mask it (streaming)?

Define: Multi-agent coordination adds latency, since each hop between agents is itself a round trip; from the user's perspective this can mean a long silent wait before anything useful appears. Architect: I stream intermediate progress, such as which agent is currently working and a summary of what it just found, to the user interface as it happens, rather than waiting for the entire multi-agent pipeline to finish before showing anything. Trade-offs: Streaming intermediate state gives the user a sense of progress and reduces perceived wait time, but it means surfacing partial, sometimes-wrong intermediate findings that might get revised later by a downstream agent. Fail-safe: Showing an intermediate finding that later turns out wrong, without updating the user, leaves a stale wrong impression; I explicitly update or retract any streamed intermediate claim that a later agent in the pipeline overturns, rather than only ever appending new information.

How do you implement dynamic agent spawning (map-reduce over subtasks)?

Define: Dynamic spawning breaks one task into an a-priori-unknown number of independent subtasks, runs an agent instance per subtask in parallel, the map step, and then combines their results into one final answer, the reduce step. Architect: A coordinator first determines how many subtasks the input actually decomposes into, spawns one agent instance per subtask with only the slice of input it needs, runs them concurrently, and a reduce step then merges their individual outputs, resolving any overlaps or conflicts between them. Trade-offs: Map-reduce over subtasks parallelizes work that would otherwise be processed one item at a time, cutting total latency, but it multiplies the total token cost by the number of subtasks and adds a non-trivial reduce step to merge potentially inconsistent results. Fail-safe: An unbounded number of spawned subtasks can exhaust budget or compute if the input is larger than expected; I cap the maximum number of spawned instances and have the coordinator batch or chunk an oversized input rather than spawning one instance per item unconditionally.

When is a multi-agent setup pure over-engineering?

Define: Multi-agent is over-engineering when the task is genuinely linear or simple enough that one agent with the right tools would have handled it just as well, but it's been split into several agents anyway. Architect: I check this by asking whether a single-agent version of the task has actually been tried and shown a concrete limitation, tool overload, conflicting personas, a genuine need for parallel work; if that test was skipped and multiple agents were chosen by default, that's the sign of over-engineering. Trade-offs: Defaulting to multi-agent to be safe avoids ever hitting a single-agent limitation, but it pays the full cost, latency, and debugging complexity of coordination on tasks that never needed it. Fail-safe: The clearest symptom in practice is a multi-agent system where the agents never actually disagree, hand off conflicting information, or do genuinely parallel work; when that's the case, I collapse it back to a single agent rather than leaving the unnecessary coordination in place.

Compare LangGraph, CrewAI, AutoGen, and OpenAI Agents SDK—when would you pick each?

Define: These are four different frameworks for building agentic systems, differing mainly in how explicitly they expose control flow: LangGraph as an explicit state graph, CrewAI around role-based crews of agents, AutoGen around conversational multi-agent chat, and the OpenAI Agents SDK as a lighter, more minimal orchestration layer tied closely to the provider's own API. Architect: I pick LangGraph when I need precise, auditable control over branching and state, especially with cycles and human-in-the-loop interrupts; CrewAI when the task naturally maps to a small set of named roles collaborating on a shared goal; AutoGen when the task benefits from open-ended conversational back-and-forth between agents; and the Agents SDK when I want a thin, low-abstraction layer and I'm already committed to that provider. Trade-offs: More explicit frameworks like LangGraph give better control and debuggability at the cost of more code to write; more opinionated frameworks like CrewAI get a working system faster but hide more of the control flow, making deep customization or debugging harder when something goes wrong. Fail-safe: Picking a heavily opinionated framework and later needing control it doesn't expose is a real risk; I prototype the riskiest, most custom part of the workflow outside the framework first to confirm it's actually expressible, before committing the whole system to it.

Why are directed cyclic graphs (like LangGraph) favored over pure linear chains for agents?

Define: A directed cyclic graph allows control flow to loop back to an earlier node, which a pure linear chain, where each step runs exactly once in a fixed order, structurally cannot do. Architect: Most real agent behavior, retrying a failed step, re-planning after new information, looping until a condition is met, requires revisiting a node more than once, which a graph with cycles expresses directly as an edge back to an earlier node, where a linear chain would need to be unrolled in advance for every possible number of iterations. Trade-offs: Cyclic graphs model realistic agent behavior far more naturally, but they also make runaway loops possible in a way a strictly linear chain cannot, so they need their own safeguards against infinite cycling. Fail-safe: Every cycle in the graph needs an explicit exit condition or iteration cap; I treat an unbounded cycle as a bug by default and require every loop-back edge to be paired with a termination check before it ships.

What are the hidden abstractions or debugging pains of off-the-shelf agent frameworks?

Define: A hidden abstraction is framework behavior happening automatically behind the scenes, such as automatic retries, implicit prompt formatting, or internal state management, that isn't visible in the code you actually wrote. Architect: When something goes wrong, I first check whether the framework is doing something automatically that isn't obvious from my own code, by reading its actual source for the relevant step or turning on its most verbose tracing, rather than assuming the bug is purely in my own logic. Trade-offs: Frameworks save substantial development time by handling common patterns automatically, but that same automation is exactly what makes a failure hard to localize, since the actual behavior causing a bug might be several layers below the code the developer wrote. Fail-safe: The specific risk is trusting a framework's default behavior without verifying what it actually does; for anything safety- or cost-critical, I read the framework's own source for that specific behavior at least once, rather than relying on its documentation's description alone.

How do you implement human-in-the-loop interrupts and state resumption in a graph workflow?

Define: A human-in-the-loop interrupt pauses a graph-based workflow at a defined node, waits for a person to approve, reject, or edit something, and then resumes execution from exactly that point once the human responds. Architect: I design the graph with an explicit interrupt node before any action that needs human approval; the workflow's state is checkpointed at that node, execution pauses and surfaces the pending decision to a human interface, and on approval the graph resumes from the saved checkpoint rather than restarting the whole workflow. Trade-offs: Building explicit interrupt points gives precise control over exactly where a human must weigh in, but it requires the workflow's state to be fully serializable at every such point, which constrains what can live only in local, unsaved variables. Fail-safe: A workflow that can't actually resume cleanly from its checkpoint after a long human delay is a real failure mode; I test resumption after a realistic delay, not just immediately after pausing, since external state the workflow depends on can have changed in between.

What role do state machines play in constraining LLM autonomy?

Define: A state machine defines a fixed set of allowed states and the specific transitions permitted between them, which constrains an agent to only move through pre-approved paths rather than taking any action its own reasoning happens to suggest. Architect: I model the task's allowed states and transitions explicitly, and let the LLM decide which of the currently valid transitions to take at each point, rather than letting it freely decide an arbitrary next action; any action not corresponding to a valid transition from the current state is simply rejected. Trade-offs: A state machine gives strong, auditable guarantees about what an agent can and can't do, which matters a lot for anything with real-world consequences, but it also limits the agent to the states and transitions a designer anticipated in advance. Fail-safe: The failure case a state machine specifically prevents is the infinite conversational drift that free-form agent loops are prone to; I use deterministic state machines exactly for the actions where unconstrained autonomy would be unacceptable, even if it means less flexibility elsewhere.

How do you structure a clean codebase for a multi-agent application?

Define: A clean multi-agent codebase separates each agent's own logic, the shared state schema, the orchestration or routing layer, and the tool implementations into distinct, independently testable modules, rather than mixing them together. Architect: I keep each agent's prompt, role definition, and decision logic in its own module, define the shared state schema once in a single place every agent imports rather than redefining it, put orchestration, who runs next, in a layer separate from any individual agent's own code, and keep tool implementations independent of any specific agent so they can be reused or tested alone. Trade-offs: This separation takes more upfront structure to set up than just writing one big script, but it pays off directly when debugging, since a failure can be isolated to one module rather than searched for across a single entangled file. Fail-safe: The most common structural mistake is letting the orchestration logic and an individual agent's own logic blend together, so a change to one agent silently affects routing; I enforce that boundary with a clear interface, defined inputs and outputs, between every agent and the orchestrator, not implicit shared variables.

Level 9: Evaluation and observability

Evaluation and observability — interview questions

Answer in your head first, then open it.

Why is evaluating an AI agent harder than evaluating a single-turn chatbot?

Define: A single-turn chatbot's evaluation only needs to judge one response to one input; an agent's evaluation has to judge an entire multi-step trajectory, where an early wrong step can still accidentally lead to a correct final answer, or a correct process can still end in a wrong final answer. Architect: I evaluate both the final outcome and, separately, the path taken to get there, since judging only the final answer can hide a process that happened to work by luck and wouldn't generalize, where judging only the trajectory can miss that the final answer was actually wrong despite a reasonable-looking process. Trade-offs: Trajectory-level evaluation gives far more diagnostic signal about where things actually go wrong, but it's far more expensive to build and label than scoring a single response, since every step along a potentially long path needs its own judgment. Fail-safe: Relying only on end-outcome evaluation risks shipping an agent that got lucky on the test set through a flawed process that fails differently on new inputs; I require trajectory review on at least a sample of passing cases, not only the failing ones.

What is the difference between evaluating the final outcome vs. evaluating the trajectory (path)?

Define: Outcome evaluation judges whether the final answer or action was correct; trajectory evaluation judges whether the sequence of steps the agent took to get there was itself reasonable, efficient, and safe. Architect: I score outcomes against a known correct answer or acceptance criteria, and separately score trajectories against criteria like whether the right tools were called, whether unnecessary steps were taken, and whether any step violated a safety rule, even if the final answer happened to be correct anyway. Trade-offs: Outcome evaluation is simpler to automate and score, since there's often a clear right answer to check against, but it can't catch a dangerous or wasteful process that happened not to cause visible harm this particular time. Fail-safe: A correct outcome reached through an unsafe step, such as an unauthorized write that happened to not break anything this time, is exactly the case outcome-only evaluation misses; I treat any trajectory-level safety violation as a failure regardless of what the final outcome looked like.

How do you build a golden dataset for testing multi-step agents?

Define: A golden dataset is a curated set of realistic tasks paired with verified correct outcomes, and ideally expected trajectories, that serves as the ground truth an agent's behavior gets measured against. Architect: I source tasks from real production logs or representative scenarios rather than inventing unrealistic ones, have a human expert verify and annotate the correct final answer and, for at least a meaningful subset, the acceptable tool-call sequence, and keep the dataset versioned so changes to it are tracked over time rather than silently drifting. Trade-offs: A dataset built from real production cases is far more representative than a synthetic one, but sourcing and annotating real cases is slower and more expensive than generating synthetic examples at scale. Fail-safe: A golden dataset that never gets updated silently becomes stale as the product and its edge cases evolve; I treat it as a living artifact with a regular review cadence, not a one-time deliverable, since an outdated golden set gives false confidence.

How do you implement LLM-as-a-Judge reliably, and how do you avoid position/verbosity bias?

Define: LLM-as-a-Judge uses a model to score or compare agent outputs against criteria, which is useful for scaling evaluation beyond what human review can cover, but it comes with known biases, such as favoring whichever answer appears first, position bias, or favoring longer answers regardless of quality, verbosity bias. Architect: I randomize the order of compared outputs across repeated judgments and average the results to cancel out position bias, give the judge an explicit rubric with concrete criteria rather than an open-ended which-is-better question, and periodically validate the judge's scores against a human-labeled sample to catch verbosity or other systematic bias. Trade-offs: LLM-as-a-Judge scales far beyond what human reviewers could cover, but it inherits its own biases and blind spots, so it's a complement to human review on a sample, not a full replacement for it. Fail-safe: An uncorrected position or verbosity bias silently skews every automated comparison in the same direction; I run the bias-detection check, randomized order, length-controlled comparisons, as a standing part of the judge's own evaluation, not a one-time calibration.

What metrics do you track for tool-calling accuracy (tool selection, parameter accuracy, call count)?

Define: Tool-calling accuracy breaks into whether the right tool was chosen for the task, whether the arguments passed to it were correct, and whether the number of calls made was appropriate, three related but distinct failure points. Architect: I measure tool-selection accuracy against a labeled set of tasks with known correct tools, measure parameter accuracy by checking whether the arguments passed match the expected ground-truth values for that task, and track call count against an expected range, flagging both unusually high counts, looping, and unusually low counts, skipped necessary steps. Trade-offs: Tracking all three separately gives precise diagnostic signal about which specific failure mode is occurring, but it requires ground-truth labels for the correct tool, correct parameters, and expected call count for every task in the evaluation set, which is more annotation work than a single pass or fail label. Fail-safe: An agent with perfect tool selection but consistently wrong parameters would look fine on tool-selection metrics alone; I always report all three metrics together rather than any single one in isolation, since any one of them passing doesn't guarantee the others did.

How do you run deterministic unit tests on non-deterministic agent code (mocking tools & LLMs)?

Define: Deterministic unit testing of agent code means testing the agent's logic, routing, and tool-handling in isolation from the LLM's own non-deterministic output, by substituting a fixed, scripted response in place of the real model call. Architect: I mock the LLM call to return a fixed, pre-defined response for a given test input, and mock each tool call to return a fixed result rather than hitting the real external system, so the test exercises the agent's own control-flow code, how it reacts to a given model output or tool result, with fully predictable inputs. Trade-offs: Mocking makes unit tests fast, free, and perfectly repeatable, but it only tests the agent's code around the model and tools, not whether the real model would actually produce the response being mocked, so it needs to be paired with separate evaluation against the real model. Fail-safe: A test suite that's entirely mocked can pass completely while the real model's actual behavior has drifted away from what the mocks assume; I run a smaller set of tests against the real model periodically specifically to catch that drift, rather than relying on mocked tests alone forever.

What is pass@k vs. pass^k (consistency across repeated runs) for agents?

Define: Pass@k measures whether at least one of k independent attempts at a task succeeds, useful for tasks where any single success counts; pass^k measures whether all k independent attempts succeed, which is the relevant metric when consistency across repeated runs, not just occasional success, is what matters. Architect: I run the same task k times independently with the same input, and for pass@k check whether any run succeeded, while for pass^k check whether every run succeeded, choosing which metric matters based on whether the real deployment lets a user retry, pass@k, or needs to work correctly the very first time, every time, pass^k. Trade-offs: Pass@k paints an optimistic picture for a task users might retry on failure, but pass^k is the honest metric for anything that needs to be reliable on a single attempt, and the gap between the two numbers reveals exactly how much hidden inconsistency pass@k alone would hide. Fail-safe: Reporting only pass@k for a task that actually needs single-attempt reliability overstates how production-ready the agent really is; I report both numbers together whenever reliability on a single attempt actually matters for the use case.

How do you simulate users to test multi-turn conversational agents at scale?

Define: User simulation uses another model, prompted to play a realistic user persona with a specific goal, to carry on a multi-turn conversation with the agent under test, generating far more test conversations than a human tester could write by hand. Architect: I define a set of distinct user personas and goals, a frustrated customer, a confused new user, an adversarial tester, have a simulator model carry on a full multi-turn conversation with the agent under test for each, and score the resulting transcripts against the same criteria used for real conversations. Trade-offs: Simulated users generate test coverage at a scale hand-written test conversations never could, but a simulator model's behavior can itself be unrealistic in ways real users aren't, so simulated results need occasional validation against real user transcripts. Fail-safe: An agent that's been tuned heavily against its own simulated users can overfit to patterns the simulator produces that real users don't, inflating apparent performance; I keep a held-out set of real user transcripts specifically to check that simulated-test performance actually generalizes.

How do you detect regressions when you upgrade the underlying base LLM or tweak a system prompt?

Define: A regression here means a change to the base model or system prompt that was intended as neutral or an improvement instead makes some previously working case fail. Architect: I run the full golden dataset and a representative trajectory-evaluation suite against both the old and new model or prompt version, side by side on the identical set of tasks, and specifically flag any case that passed before and fails now, rather than only checking whether the overall aggregate score moved. Trade-offs: Case-by-case comparison catches a regression hidden inside an overall score that happened to stay flat or even improve on average, but it requires keeping the full evaluation set and its prior results around for comparison, not just a single headline number from the last run. Fail-safe: An aggregate score that improved overall can still hide a serious regression on a smaller but important subset of cases; I weight or separately track any safety-critical or high-frequency case category so it can't be silently offset by unrelated gains elsewhere in the average.

What public benchmarks (e.g., SWE-bench, GAIA, Tau-bench, OSWorld) measure agent capabilities?

Define: These are standardized, published benchmarks each measuring a different slice of agent capability: SWE-bench measures real-world software-engineering task completion, GAIA measures general multi-step reasoning and tool use on broad assistant tasks, Tau-bench measures an agent's ability to follow domain-specific policies in simulated customer-service-style interactions, and OSWorld measures an agent's ability to actually operate a real computer desktop environment. Architect: I use these to benchmark a general capability or compare against a published baseline, but I still rely primarily on the organization's own golden dataset for whether the agent is ready for its actual production task, since a public benchmark measures a generic task distribution, not necessarily the one the agent will actually face in production. Trade-offs: Public benchmarks let you compare against the broader field and track how a model or framework is improving over time, but a strong benchmark score doesn't guarantee strong performance on the organization's own specific, narrower task, since benchmark tasks are necessarily generic. Fail-safe: Treating a public benchmark score as a substitute for the organization's own evaluation is the specific risk; I use public benchmarks for comparison and marketing context only, never as the actual go or no-go signal for shipping a specific production agent.

How do you measure task completion rate vs. cost-per-successful-task?

Define: Task completion rate measures what fraction of attempted tasks the agent actually completes successfully; cost-per-successful-task measures the total resource spend, tokens, tool calls, compute, divided only by the successful completions, which together reveal whether a high completion rate was bought at an unsustainable cost. Architect: I track both metrics side by side for every change to the agent, since a change that raises completion rate by adding more retries or reflection rounds will also raise cost-per-successful-task, and a change that lowers cost by skipping steps might lower completion rate too, so neither metric alone tells the full story. Trade-offs: Optimizing for completion rate alone can justify arbitrarily expensive strategies, more retries, bigger models, more reflection rounds, that aren't actually sustainable at production volume; optimizing for cost alone can justify cutting corners that quietly reduce how often the task actually gets done right. Fail-safe: A team incentivized on completion rate alone will tend to keep adding expensive safety nets without noticing the cost curve; I report both metrics together on every dashboard so a completion-rate win can't be celebrated in isolation from what it cost to achieve.

How do you conduct shadow testing or A/B testing for a new agent version in production?

Define: Shadow testing runs a new agent version alongside the live one on real production traffic without its output actually being shown to or acted on by users, purely to compare behavior; A/B testing actually serves the new version's real output to a subset of real users and compares outcomes. Architect: For shadow testing, I route a copy of real incoming requests to the new version, log its outputs and decisions, and compare them against the live version's actual outputs without the new version's results ever reaching a user; once shadow results look acceptable, I move to A/B testing a small percentage of real traffic with real outcome tracking before a full rollout. Trade-offs: Shadow testing carries zero risk to real users since its output never reaches them, but it also can't measure real user reactions to the new version's actual behavior, which only A/B testing with real traffic can reveal. Fail-safe: A new version that looks fine in shadow testing can still behave differently once its actions have real consequences a user reacts to; I never skip the A/B phase on the assumption that a clean shadow test alone is sufficient for a full rollout.

How do you evaluate hallucination rates when an agent synthesizes data from multiple tools?

Define: Hallucination here means the agent's final synthesized answer contains a claim not actually supported by any of the tool outputs it gathered, which is harder to catch than single-source hallucination since the claim might plausibly look like it came from combining several real sources. Architect: I decompose the final answer into individual claims and check each one against the full set of tool outputs the agent actually received, flagging any claim that isn't directly traceable to at least one of those outputs, rather than just checking whether the overall answer sounds plausible given the general topic. Trade-offs: Claim-level attribution checking catches synthesis hallucination that a holistic plausibility check would miss, but decomposing and checking every claim against every tool output is significantly more expensive to automate than a single overall judgment call. Fail-safe: A claim that blends two real facts from two different tools into a third, false combined claim is the specific failure this kind of hallucination produces; I check not just whether each individual fact used appeared somewhere in the tool outputs, but whether the specific combination or inference drawn from them was actually supported.

What is online evaluation vs. offline evaluation in an agent CI/CD pipeline?

Define: Offline evaluation runs the agent against a fixed, pre-built dataset before any deployment, entirely separate from live traffic; online evaluation measures the agent's actual behavior and outcomes on real production traffic after it's already deployed. Architect: I gate every deployment on passing offline evaluation against the golden dataset first, since that catches known regressions cheaply before anything reaches a real user, and then continue monitoring the same classes of metrics online after deployment, since the real production traffic distribution will eventually diverge from whatever the offline dataset captured. Trade-offs: Offline evaluation is cheap, fast, and safe to run on every single change, but it can only catch problems the dataset's authors anticipated; online evaluation catches real, previously-unanticipated failure modes, but only after they've already happened to real users. Fail-safe: Relying on offline evaluation alone gives false confidence once production traffic drifts away from the dataset's original distribution; I feed interesting or failing online cases back into the offline golden dataset on an ongoing basis, so the two evaluation stages actually reinforce each other over time rather than staying static and separate.

Level 10: Safety, security and governance

Safety, security and governance — interview questions

Answer in your head first, then open it.

What is indirect prompt injection, and how do you protect an agent reading untrusted web pages or emails?

Define: Indirect prompt injection is malicious instruction text hidden inside content the agent reads as data, such as a web page or an email body, that attempts to get the model to treat it as a command rather than as untrusted information to merely read. Architect: I structurally separate untrusted content from the agent's actual instructions, wrapping anything read from an external, untrusted source in a way the model is explicitly told to treat as data, and I never let a tool call triggered by reading untrusted content proceed without passing through the same permission and confirmation checks any other action would require. Trade-offs: Treating all external content as inherently untrusted and gating any resulting action behind normal permission checks closes off most injection paths, but it means even a genuinely useful instruction embedded in a legitimate document can't be acted on automatically without the same scrutiny as a malicious one. Fail-safe: The actual backstop isn't detecting the injection text itself, since that's an arms race; it's that any privileged or irreversible action the model might be tricked into attempting still has to clear a deterministic permission boundary that the model's own text can't talk its way past.

How do you enforce the Principle of Least Privilege for agent tool credentials?

Define: Least privilege means every tool an agent can call is scoped to only the specific permissions that task genuinely needs, not the broadest credentials available, so a compromised or misled agent can only do limited damage. Architect: I issue scoped, short-lived credentials per session or per task rather than one standing broad credential shared across everything the agent does, and I define separate, narrower tool permissions for read access versus write access rather than one tool with blanket access to an entire system. Trade-offs: Finely scoped credentials take more upfront work to design and issue correctly for every distinct task, but a broad standing credential, if the agent is ever misled or compromised, exposes everything that credential can touch rather than just what the current task actually needed. Fail-safe: A credential that's broader than the current task needs is exactly what turns a contained mistake into a serious incident; I audit tool credential scopes periodically against what each tool actually uses in practice, tightening any that are broader than necessary.

When must you require Human-in-the-Loop (HITL) approval before an action executes?

Define: HITL approval is required specifically before any action that's high-risk or genuinely irreversible, where a wrong automated decision would cause real, hard-to-undo harm. Architect: I classify every tool by its blast radius and reversibility, require explicit human approval before executing anything above a defined risk threshold, such as sending an external communication, deleting data, or moving money, and let anything below that threshold, like a read-only lookup, execute automatically without a human gate. Trade-offs: Requiring human approval on every high-risk action slows down the agent and reintroduces a human bottleneck exactly where automation was supposed to help, but automating an irreversible action without a check trades that delay for the risk of a mistake that can't be undone. Fail-safe: The actual rule I apply is that if an action can't be cleanly undone and getting it wrong would be costly, it gets a human gate regardless of how confident the model's reasoning looked, since model confidence and actual correctness aren't reliably the same thing.

How do you prevent data exfiltration through markdown image links or unauthorized API parameters?

Define: This is a specific exfiltration technique where a model is tricked into embedding sensitive data into a URL, such as a markdown image link, that then gets silently fetched by a rendering client, sending that data to an attacker-controlled server as a side effect of just displaying the response. Architect: I strip or sanitize any model-generated markdown image or link URLs before they're rendered to a user, specifically blocking any URL that points outside an explicit allowlist of trusted domains, and I validate every outbound API call's parameters against expected values rather than letting the model construct an arbitrary URL or parameter freely. Trade-offs: An allowlist of trusted domains for any rendered link closes off this specific exfiltration path, but it also blocks any legitimate link to a domain not yet on the allowlist until it's explicitly added, trading some flexibility for the security guarantee. Fail-safe: Even with sanitization, I treat any case where the model's own output could trigger a network request as worth monitoring; I log and alert on outbound requests triggered by rendering model output, so an exfiltration attempt that slipped past sanitization still gets caught in the act.

How do you sandbox code execution when an agent writes and runs Python/Bash scripts?

Define: Sandboxing means running agent-generated code in an isolated environment that can't reach or affect anything outside itself, regardless of what the code actually tries to do. Architect: I run generated code in a container or microVM with no network access beyond an explicit allowlist, no access to the host filesystem beyond a scoped, disposable working directory, and a hard CPU, memory, and wall-clock time limit, tearing the sandbox down completely after each execution rather than reusing it across runs. Trade-offs: A fully isolated, disposable sandbox gives strong containment guarantees, but it adds real infrastructure cost and startup latency compared to just running code directly in the host process. Fail-safe: The risk that actually matters is a sandbox escape, where isolated execution turns out not to be as isolated as assumed; I treat sandbox isolation as something to be independently security-tested and kept patched, not as a one-time setup that stays safe indefinitely.

What are input guardrails vs. output guardrails, and where do they sit in the agent loop?

Define: Input guardrails check and filter what goes into the model, such as detecting a prompt injection attempt or a policy-violating request, before the model ever processes it; output guardrails check what the model produces, such as detecting a harmful or policy-violating response, before it's acted on or shown to a user. Architect: I run input guardrails as the very first step on any incoming request or tool observation, before it reaches the model's context at all, and run output guardrails as the very last step on anything the model produces, before it's rendered to a user or passed to an action-executing tool, so both ends of the loop get an independent check. Trade-offs: Running both input and output guardrails catches more than either alone, since a request can be benign but produce a harmful response, or a request can itself be the attack even if the model's response to it looks fine, but each guardrail adds its own latency and can introduce its own false positives that block legitimate requests. Fail-safe: Relying on only one side is the specific gap; an attack that passes a clean-looking input but produces a harmful output would be missed by input guardrails alone, so I treat the two as independent, non-substitutable layers rather than either one standing in for the other.

How do you strip or mask PII (Personally Identifiable Information) before sending context to third-party LLMs?

Define: PII stripping or masking removes or obscures personally identifying data, such as names, emails, or account numbers, before any text containing it is sent to an external model provider that the organization doesn't fully control. Architect: I run a detection pass, typically a combination of pattern matching for structured identifiers and a classifier for unstructured mentions, over any content before it leaves the organization's own boundary, replacing detected PII with a placeholder token, and reversing that substitution only after the response comes back, within the trusted boundary. Trade-offs: Masking before sending protects against a third-party provider ever seeing raw sensitive data, but detection isn't perfect, and overly aggressive masking can degrade the model's ability to reason about the content if too much gets replaced with placeholders. Fail-safe: A missed PII detection means raw sensitive data still leaves the boundary despite the safeguard; I treat detection recall, not just precision, as the metric that matters most here, since a missed detection is a real data exposure while an overly cautious false positive is just an inconvenience.

How do you maintain an immutable audit trail for compliance (SOC2, HIPAA, GDPR) of what an agent did and why?

Define: An immutable audit trail is a tamper-evident, append-only record of every action an agent took, the reasoning or input that led to it, and when it happened, kept specifically so a compliance review can reconstruct exactly what happened and why, after the fact. Architect: I log every tool call, its arguments, its result, and the model's stated reasoning at the time, to a write-once storage system that the application itself can't later modify or delete from, tagged with the user, session, and timestamp, rather than relying on mutable application logs that could be altered or rotated away. Trade-offs: Write-once, tamper-evident logging adds real storage cost and some engineering overhead compared to normal rotating application logs, but normal logs don't satisfy a compliance requirement that specifically demands the record can't have been altered after the fact. Fail-safe: A gap in the audit trail, where an action happened but wasn't logged before a crash, is itself a compliance finding; I log the intent to act before executing, not only the result afterward, so even a crash mid-action leaves a record of what was attempted.

What is confused deputy privilege escalation in multi-user agents?

Define: A confused deputy problem is when an agent, acting with its own broad, trusted credentials on behalf of a lower-privileged user, can be tricked into performing an action using its own elevated privileges that the requesting user was never actually authorized to perform themselves. Architect: I have the agent act using the requesting user's own permission scope on every action, not a single shared service credential with broader access than any individual user has, so the agent can never do more on a user's behalf than that specific user could already do directly. Trade-offs: Passing through the actual user's permission scope on every action is more work to implement than using one convenient shared service credential for everything the agent does, but the shared-credential approach is exactly what makes the confused deputy problem possible in the first place. Fail-safe: The specific exploit path is a lower-privileged user crafting a request that causes the agent to use its own elevated access rather than the user's actual access; I validate on every action that the resulting permission check used the requesting user's scope, not the agent's own service-level credential, as the final authority.

Level 11: Production architecture and system design

Production architecture and system design — interview questions

Answer in your head first, then open it.

When do you use synchronous vs. asynchronous agent execution?

Define: Synchronous execution has the user wait for the agent to finish before getting any response; asynchronous execution lets the agent work in the background and notify the user or write results somewhere once it's done. Architect: I use synchronous execution for short tasks where a user is actively waiting and the whole task completes within an acceptable response window, and asynchronous execution for anything that could run long, minutes or more, routing the task to a background job queue and surfacing progress or a completion notification rather than holding a connection open. Trade-offs: Synchronous execution is simpler to build and reason about, but it caps how long a task can take before it becomes an unacceptable user experience; asynchronous execution removes that cap but adds real complexity around job tracking, notification, and resuming a user's context once the result is ready. Fail-safe: A synchronous request that unexpectedly runs long, because a task turned out more complex than assumed, risks timing out and losing the work entirely; I set an explicit cutoff where a synchronous task that's taking too long gets converted to an async job rather than timing out with nothing to show for the work already done.

What makes an agent "production-ready" versus a prototype?

Define: A production-ready agent has the operational guarantees a prototype typically skips: error handling for every failure mode, cost and latency bounds, monitoring, and a tested recovery path, not just correct behavior on the happy path. Architect: I check that the agent has idempotent handling for every mutating action, a hard cap on steps and spend, structured tracing for every decision, tested fallback behavior for every external dependency it relies on, and an evaluation suite run before every change, none of which a quick prototype usually has in place. Trade-offs: Building all of this production scaffolding takes real additional time beyond getting the happy path working, but shipping without it means the first unexpected failure in production has no safety net and no way to diagnose what happened. Fail-safe: The clearest sign something isn't production-ready is that its behavior under failure, a timeout, a malformed tool response, a budget overrun, has never actually been tested; I specifically test failure injection, not just the happy path, before calling anything production-ready.

How do you build a lightweight production agent without using heavy orchestration frameworks?

Define: A lightweight agent implements just the core loop, observe, decide, act, repeat, directly in application code, using the model provider's API and a hand-written state object, without adopting a full orchestration framework's abstractions. Architect: I write the loop explicitly: call the model with the current state and available tools, execute whatever tool call comes back, append the result to state, and repeat until a stop condition, keeping the entire control flow visible in a small amount of own code rather than delegated to a framework's internals. Trade-offs: A hand-written loop is fully transparent and easy to debug since there's no hidden framework behavior, but it means re-implementing things a framework would have given for free, like checkpointing or multi-agent handoff, if the system later needs them. Fail-safe: Starting lightweight is the safer default specifically because the cost of outgrowing it, adding a framework later, is much lower than the cost of debugging through unwanted framework abstractions on a project that never actually needed them.

How do durable execution engines (like Temporal or Inngest) pair with AI agents?

Define: A durable execution engine guarantees that a long-running workflow's progress survives crashes, restarts, and failures, automatically resuming exactly where it left off, which pairs naturally with an agent loop that can run for many steps over an extended period. Architect: I wrap the agent's step-by-step loop as a durable workflow, where each step, a model call, a tool call, is a durable activity the engine automatically retries on failure and whose result is persisted once it succeeds, so a crash mid-trajectory resumes from the last completed step rather than restarting the whole task. Trade-offs: A durable execution engine removes the need to hand-build checkpointing and retry logic, but it adds another infrastructure dependency and a specific programming model the whole workflow has to be written against. Fail-safe: The main risk is forgetting that a durable activity needs to be idempotent, the same requirement covered for tool calls generally, since the engine's automatic retries will re-run an activity that failed in an ambiguous way, and a non-idempotent one would duplicate its effect on retry.

How do you stream intermediate agent thoughts and tool statuses to a frontend UI?

Define: Streaming intermediate state means sending partial updates, such as which tool is currently running or a chunk of the model's reasoning as it's generated, to the user interface as they happen, rather than waiting for the entire agent run to finish before sending anything. Architect: I use a streaming protocol, server-sent events or a WebSocket, to push discrete event types, such as tool-started, tool-finished, and partial text tokens, from the backend to the frontend as the agent loop progresses, with the frontend rendering each event type with its own appropriate UI treatment rather than just appending raw text. Trade-offs: Streaming gives the user a much better sense of progress and reduces perceived latency on long-running tasks, but it means the backend and frontend both need to handle a continuous event stream and partial, possibly-later-superseded state, which is more complex than a single request-response cycle. Fail-safe: A dropped connection mid-stream can leave the frontend showing a stuck, incomplete state with no indication anything went wrong; I pair streaming with a heartbeat or timeout on the frontend that surfaces a clear error if updates stop arriving, rather than leaving the UI silently frozen.

How do you handle model routing (e.g., using a fast/cheap model for routing and a frontier model for planning)?

Define: Model routing uses a cheaper, faster model for simple, high-volume decisions, such as classifying which path a request should take, and reserves a more expensive, more capable model for the genuinely hard reasoning steps, rather than using one model tier for every step regardless of difficulty. Architect: I classify each step in the agent's workflow by how much reasoning capability it actually needs, route simple classification or extraction steps to a small, cheap model, and route complex planning, multi-step reasoning, or final synthesis steps to a frontier model, measuring whether the cheaper model's accuracy on its assigned steps is actually acceptable before committing to the split. Trade-offs: Routing cuts the average cost per request substantially since most steps in a typical trajectory are simple, but it requires correctly identifying which steps are actually simple enough for a cheaper model, and a wrong classification sends a hard step to a model that can't handle it well. Fail-safe: A cheap model silently producing a lower-quality result on a step that turned out harder than expected is worse than an obvious failure; I monitor the cheap model's output quality specifically on its routed steps, not just overall system accuracy, so a degrading split gets caught rather than averaged away.

How do you monitor and trace multi-step agent runs in production (e.g., LangSmith, Arize Phoenix, OpenTelemetry)?

Define: Tracing captures the full sequence of steps, model calls, tool calls, and their inputs and outputs, for a single agent run, so a specific run's behavior can be inspected after the fact rather than inferred from aggregate metrics alone. Architect: I instrument every model call and tool call with a tracing library that tags each span with the shared run ID, capturing inputs, outputs, latency, and token usage for each step, and send that trace data to an observability platform built for this shape of data, so a specific failing run can be opened and walked through step by step. Trade-offs: Full tracing on every run gives complete visibility for debugging, but it adds real overhead in storage and some latency, so a high-volume production system might need to sample traces rather than capture every single one. Fail-safe: Sampling traces risks missing exactly the rare failure case that most needed investigation; I always capture full traces for any run that errors or gets flagged by an evaluation check, even when ordinary successful runs are sampled, so the cases that matter most are never the ones missing data.

What specific telemetry spans do you log for every agent step?

Define: A telemetry span is a timed record of one discrete unit of work within a run, and for an agent step that means capturing enough detail to reconstruct exactly what the model or tool did and how long it took. Architect: For each model call I log the prompt, or a reference to it, the response, token counts, latency, and model version; for each tool call I log the tool name, arguments, result, latency, and success or failure status; and I tag every span with the run ID and step number so spans can be reassembled into the full trajectory in order. Trade-offs: Logging full prompts and responses gives the richest debugging detail, but it can capture sensitive data that then has to be handled under the same retention and access rules as the original data, which is more overhead than logging only metadata. Fail-safe: Logging only metadata, latency, success flag, without the actual content is cheaper but useless for understanding why a specific run went wrong; I always log full content for at least failed or flagged runs, even if successful runs are logged more sparsely to manage volume.

How do you set budgets, token caps, and max-iteration limits to prevent runaway bills?

Define: These are hard limits set before a run starts that cap how much a single agent run, or a single user's aggregate usage, can spend or iterate, specifically to bound the worst case regardless of what the agent's own reasoning decides to do. Architect: I set a maximum token budget and a maximum number of loop iterations per run, enforced by the orchestration code itself rather than left to the model's own judgment about when to stop, and separately track aggregate spend per user or per tenant against a billing-level cap that halts further requests once exceeded. Trade-offs: Hard caps guarantee the worst case is bounded, but a cap set too low can cut off a legitimate task partway through, while a cap set too generously doesn't actually protect against the runaway-cost scenario it was meant to prevent. Fail-safe: The cap itself is the fail-safe for a model that doesn't reliably stop on its own; I treat hitting the cap as a signal to investigate why the task didn't finish within the expected budget, not just a silent cutoff, since a well-designed task hitting its cap regularly usually means the cap or the task design needs adjusting.

How do you reduce end-to-end latency in a 6-step agent loop?

Define: Reducing end-to-end latency in a multi-step loop means cutting the total wall-clock time across all six steps, which requires looking at both each step's own latency and whether steps that don't depend on each other are actually running in parallel. Architect: I profile each of the six steps individually to find which ones dominate total latency, parallelize any steps that don't depend on each other's results, use prompt caching for any stable prompt prefix repeated across steps, and consider routing simpler steps to a faster model, rather than trying to speed up every step equally. Trade-offs: Parallelizing independent steps and caching stable prefixes often yields the largest latency wins for the least behavioral risk, but routing to faster models for speed can trade away some accuracy on the steps that get switched. Fail-safe: Optimizing for latency without re-running the evaluation suite risks silently trading away accuracy for speed; I re-run the same evaluation suite after any latency optimization to confirm it didn't degrade correctness, not just that it got faster.

When would you use speculative execution or parallel branching to speed up agents?

Define: Speculative execution starts work on a likely next step before the current step has actually confirmed that path is correct, discarding the speculative work if it turns out wrong; parallel branching runs several plausible next steps at once and picks the one that turns out right, rather than waiting to decide sequentially. Architect: I use this when a step's likely outcome can be predicted with reasonable confidence before it's confirmed, starting the probable next step's work in parallel with the confirmation, and discarding the speculative branch's work, without ever surfacing it, if the prediction turns out wrong. Trade-offs: Speculative execution can meaningfully cut latency when predictions are usually right, but every wrong speculation wastes the compute spent on the discarded branch, so it only pays off when the prediction accuracy is high enough to outweigh that waste. Fail-safe: The real risk is a speculative branch's side effect leaking out before it's confirmed wrong, such as a tool call with a real-world effect fired speculatively; I only speculate on read-only or otherwise side-effect-free steps, never on anything mutating, so a wrong guess costs compute but never a real action.

How do you handle LLM provider outages or rate-limit spikes (fallback models)?

Define: A fallback strategy means the system automatically switches to an alternate model or provider when the primary one is unavailable or rate-limited, rather than simply failing the request outright. Architect: I configure a ranked list of fallback models, ideally from a different provider than the primary, detect outage or rate-limit-specific error responses distinctly from other errors, and automatically retry the same request against the next model in the fallback list rather than surfacing the failure immediately to the user. Trade-offs: Falling back to a different model keeps the system available during an outage, but different models can behave differently enough that output quality or format may shift mid-incident, so fallback responses should be flagged as using a degraded path. Fail-safe: Silently serving degraded fallback output without any record of it happening hides a real reliability event; I log every fallback activation explicitly, so the operations team can see exactly how often and for how long the system was running on its fallback path, not just that uptime looked fine overall.

How do you implement semantic caching for repetitive agent queries?

Define: Semantic caching returns a previously computed response for a new query that's semantically similar enough to an earlier one, rather than requiring an exact text match the way a traditional cache key would. Architect: I embed each incoming query, check it against a cache of previously embedded queries and their stored responses using a similarity threshold, and return the cached response directly if a sufficiently similar prior query exists, falling through to a real model call and caching the new result when nothing close enough is found. Trade-offs: Semantic caching catches far more repeat traffic than exact-match caching since users phrase the same underlying question differently, but setting the similarity threshold too loose risks returning a stale or subtly wrong cached answer for a query that wasn't actually asking the same thing. Fail-safe: A cached response served for a query that was subtly different from the original is a silent correctness risk, not just a performance tradeoff; I set the similarity threshold conservatively and specifically test it against known near-duplicate-but-different query pairs, not just genuinely identical ones.

What is the cost trade-off between fine-tuning a small model (SLM) for tool calling vs. prompting a large frontier model?

Define: Fine-tuning a small model trains it specifically on the organization's own tool-calling patterns, trading an upfront training investment for a much cheaper and faster per-call cost at inference time; prompting a frontier model needs no training investment but pays a higher per-call cost indefinitely. Architect: I choose fine-tuning when the tool-calling pattern is stable, high-volume, and narrow enough that a small model can learn it reliably, and prompting a frontier model when the task is still evolving, low enough volume that per-call cost doesn't dominate, or broad enough that a small model wouldn't generalize well across it. Trade-offs: Fine-tuning pays off at scale through much lower per-call cost, but it requires maintaining training data and a retraining pipeline as the underlying tools or patterns change, where prompting a frontier model stays current automatically as the provider improves it. Fail-safe: A fine-tuned small model silently degrading as the tools it was trained on evolve, without anyone noticing because it still runs without obvious errors, is the specific risk; I monitor the fine-tuned model's accuracy against fresh evaluation data on an ongoing basis, not just at the point it was first deployed.

How do you handle database connection pooling and state locking when concurrent requests hit the same agent thread?

Define: This is about preventing two concurrent requests from corrupting or racing on the same underlying conversation or session state, and about not exhausting database connections when many agent instances are running at once. Architect: I use a connection pool sized to the actual concurrent load rather than opening a new connection per request, and acquire an explicit lock or lease on a given session's state before any step that reads and then writes it, so two concurrent requests against the same session thread can't interleave their reads and writes into a corrupted result. Trade-offs: Locking session state prevents corruption from concurrent access, but it also means a second request against the same session has to wait or be rejected while the first is still being processed, which needs a clear, fast failure or queuing behavior rather than an indefinite block. Fail-safe: A connection pool that's too small under real load causes requests to queue or fail in a way that looks like an application bug rather than a capacity limit; I monitor pool utilization directly and alert well before it's actually exhausted, rather than discovering the limit from user-facing errors.

How do you debug an agent failure that only happens 1% of the time in production?

Define: A rare, intermittent failure needs enough captured detail on every run, not just failing ones, to reconstruct the 1% case after the fact, since it's too infrequent to reliably reproduce on demand. Architect: I rely on full tracing infrastructure to capture every run's complete trajectory, filter traces specifically for the failure signature once one is observed, look for what's different about those runs, a specific input pattern, a particular tool's response shape, a timing condition, and reproduce the failure in a controlled test using the exact captured inputs from a real failing trace rather than a guessed-at scenario. Trade-offs: Comprehensive tracing is what actually makes a 1% failure debuggable at all, but it means accepting the storage and overhead cost of capturing detailed data for the 99% of runs that never fail, purely so the rare failing 1% can be investigated. Fail-safe: Without captured traces specifically for failed runs, a rare failure can become effectively undebuggable, recurring indefinitely with no way to pin down its cause; I always guarantee full trace retention for failed or flagged runs specifically, even under aggressive sampling for everything else.

How do you manage prompt versioning and rollback in production?

Define: Prompt versioning treats every change to a system prompt or template as a tracked, versioned artifact, the same way code changes are versioned, so a specific version can be identified, compared, and rolled back to if it causes a problem. Architect: I store every prompt version with an explicit version identifier, tag every production run with which prompt version actually produced it, and keep the deployment process able to roll back to a prior prompt version immediately, the same way a code deployment would roll back, rather than editing a live prompt in place with no record of what changed. Trade-offs: Formal prompt versioning adds process overhead to what could otherwise be a quick text edit, but an untracked, in-place prompt edit means a regression caused by that edit has no clear point to roll back to and no record of what the prompt even said before. Fail-safe: An unversioned prompt change that causes a regression is especially costly because there's no fast, confident way to revert it; I never let a prompt change ship without going through the same versioned, rollback-capable path as any other production change, regardless of how small the edit looks.

What SLAs/SLOs make sense for an asynchronous background agent vs. a real-time chat agent?

Define: An SLA or SLO defines the measurable reliability and performance target a system commits to, and the right targets differ sharply between an agent a user is actively waiting on in a chat and one running a background task with no one watching in real time. Architect: For a real-time chat agent, I set tight latency SLOs, a response within a few seconds, since a user is actively waiting, with completion rate as a secondary concern; for an asynchronous background agent, I set a looser latency target, completion within a defined window like an hour, but a stricter completion-rate and correctness SLO, since there's no user to notice or retry a slow response, only a wrong or missing one. Trade-offs: Tight latency SLOs for a chat agent constrain how much reasoning or tool use it can do per turn, since every added step costs visible wait time; looser latency for a background agent allows more thorough processing, but it also means a problem can go unnoticed for the full length of that longer window before anyone checks. Fail-safe: A background agent with only a loose latency SLO and no active monitoring can fail silently for the entire window before anyone notices; I pair any loose latency SLO with proactive alerting on completion-rate and error-rate thresholds, rather than relying on someone to eventually check whether the job finished.

Design a Customer Support Refund Agent: Must check order history, verify policy rules, issue refunds up to $50 automatically, and escalate anything higher to a human.

Define: This system needs an agent that can autonomously resolve small, well-bounded refund requests while reliably routing anything larger or policy-ambiguous to a human, which is fundamentally a risk-tiered automation design, not a pure reasoning problem. Architect: The agent retrieves the order history and the relevant refund policy via read-only tools, computes whether the request is policy-eligible and under the $50 threshold, and only if both checks pass does it call the mutating refund tool, itself guarded by an idempotency key; anything over $50 or policy-ambiguous is routed to a human queue with the full context already gathered, rather than making the human start from scratch. Trade-offs: Automating only the clearly bounded, low-risk cases keeps the blast radius of a mistake small and keeps human reviewers focused on genuinely judgment-requiring cases, but it means a larger share of total refund volume by dollar value still needs a human, so the automation's efficiency gain is capped by the threshold chosen. Fail-safe: The $50 threshold itself is the fail-safe boundary, and it has to be enforced deterministically in code, not left to the model's own judgment about whether a request seems reasonable; I also idempotency-key every actual refund call so a retried decision can never double-refund the same request.

Design an Autonomous Coding / PR Review Agent: Must clone a repo, understand the codebase, run tests in a sandbox, fix bugs, and open a pull request.

Define: This agent needs to safely explore and modify a real codebase, verify its own changes actually work before proposing them, and produce a reviewable artifact, a pull request, rather than pushing changes directly, keeping a human in the loop for the final decision. Architect: The agent clones the repo into an isolated sandbox with no access to production systems, uses read tools to understand the relevant code and reproduce the reported bug, makes code changes, runs the test suite inside that same sandbox to verify the fix actually passes before proceeding, and opens a pull request rather than merging directly, so a human reviews the actual diff before it reaches the real codebase. Trade-offs: Requiring a passing test run before opening the PR catches many broken fixes automatically, but it only catches what the existing test suite actually covers, so a fix that passes tests can still be wrong in a way the suite doesn't check for. Fail-safe: Never merging directly, only opening a PR, is the structural fail-safe here; even a confidently-wrong fix that passed its own sandbox tests still has to clear human review before it reaches real users, so the sandbox's test pass is a filter, not a final authority.

Design a Deep Research Financial Analyst Agent: Must pull SEC filings, query live market APIs, verify numbers without hallucinating, and generate a cited report.

Define: This agent needs to ground every number in its final report in an actual retrieved source, since a hallucinated figure in a financial report is a specific, serious failure mode, not a generic quality issue. Architect: The agent retrieves the actual SEC filing and market data through dedicated tools rather than recalling figures from training knowledge, attaches a citation to every claimed number tracing it back to the specific retrieved source and filing, and runs a verification pass, similar to the claim-attribution checking used for RAG hallucination, that checks every number in the draft report actually traces back to a retrieved source before the report is finalized. Trade-offs: Requiring every number to trace to a specific retrieved source is more constraining and slower than letting the model freely synthesize figures from its own knowledge, but a financial report with even one hallucinated number undermines trust in the whole document. Fail-safe: A number that can't be traced to a retrieved source gets flagged or removed rather than included anyway on the assumption it's probably close enough; I treat unverifiable as its own explicit category in the report rather than silently filling the gap with the model's best guess.

Design a Multi-Tenant Enterprise Scheduling & Email Agent: Must handle calendar conflicts, external emails, prompt injection defense, and user permissions.

Define: This agent sits at the intersection of several risks covered already: it reads untrusted external email content, acts across multiple tenants whose data must stay isolated, and can take real scheduling and messaging actions, so it needs tenant isolation, injection defense, and permission scoping all operating together. Architect: Every memory and calendar lookup is scoped by tenant ID as a hard filter, any content read from an external email is treated as untrusted data never capable of directly triggering an action, and any mutating action, like sending an email or booking a meeting, runs under the requesting user's own actual permission scope rather than a shared service credential, consistent with the confused-deputy protection covered under security. Trade-offs: Enforcing tenant isolation and per-user permission scoping on every single action is considerably more implementation work than a single shared-credential, single-tenant design, but any shortcut here risks either cross-tenant data leakage or a confused-deputy privilege escalation, both serious incidents. Fail-safe: An external email successfully convincing the agent to send a calendar invite it shouldn't is the specific attack this design has to resist; I require any instruction that appears to originate from read email content to be re-confirmed against the requesting user's own actual request, never acted on directly just because it was phrased as an instruction inside the email.

Design a Self-Healing DevOps Incident Response Agent: Must read PagerDuty alerts, query logs/metrics, identify root causes, and safely execute rollback runbooks.

Define: This agent needs to diagnose a live production incident quickly while being extremely conservative about the actions it's allowed to take autonomously, since a wrong automated action during an active incident can make things measurably worse. Architect: The agent reads the alert and queries logs and metrics through read-only tools to form a root-cause hypothesis, and for remediation, it's restricted to executing only pre-approved, well-tested rollback runbooks rather than improvising a novel fix, with any action outside that known-safe runbook set requiring human approval before it executes. Trade-offs: Restricting autonomous action to known-safe runbooks means the agent can resolve the common, well-understood incident types quickly without waking anyone up, but it can't autonomously handle a genuinely novel incident that doesn't match any existing runbook, which still needs a human. Fail-safe: During an active incident, a wrong automated action is worse than a slower human-approved one, since it compounds the existing problem; every runbook action is itself idempotent and reversible where possible, and the agent is explicitly barred from chaining multiple remediation actions together without a human checking in between, so one bad step can't cascade into several.

Level 12: Agent products and patterns

Agent products and patterns — interview questions

Answer in your head first, then open it.

What is the difference between an AI agent, a copilot, and an assistant?

Define: A copilot works alongside a human inside their existing workflow, suggesting or drafting while the human stays in control of every final action; an assistant responds to direct requests and carries them out with moderate autonomy; an agent operates with the most autonomy, pursuing a goal across multiple steps with comparatively little per-step human involvement. Architect: I pick based on how much autonomy the product actually wants to grant: a copilot pattern suits a tool where the human is the one doing the work and wants suggestions, a code editor's inline completions, an assistant pattern suits a tool that takes a direct request and executes it in a bounded way, booking a specific meeting, and an agent pattern suits an open-ended goal that may take many unpredictable steps to achieve, researching and compiling a report. Trade-offs: More autonomy, moving from copilot toward agent, can get more done per user interaction, but it also means a mistake happens further from an active human check, so the right amount of autonomy is a product decision about risk tolerance, not just a capability question. Fail-safe: Granting agent-level autonomy to a task that's really better suited to a copilot or assistant pattern multiplies the blast radius of any single mistake unnecessarily; I default to the least autonomous pattern that still accomplishes the product's actual goal, rather than defaulting to the most capable-sounding one.

Level 13: Advanced topics

Advanced topics — interview questions

Answer in your head first, then open it.

How do reasoning models (like o1/o3 or DeepSeek-R1) change how we design agent loops?

Define: A reasoning model performs extended internal deliberation before producing its visible output, which changes agent loop design since much of the thinking that used to require an explicit ReAct-style thought step now happens inside the model's own reasoning process before it ever responds. Architect: With a reasoning model, I often simplify the explicit scaffolding, relying less on manually prompted step-by-step reasoning and more on the model's own internal deliberation, while still keeping the loop's structural guardrails, such as step caps and tool-call validation, since those protect against failure modes that exist regardless of how the model reasons internally. Trade-offs: Reasoning models can reduce the need for heavy manual prompting scaffolding and often improve accuracy on hard multi-step tasks, but their extended internal deliberation costs more latency and tokens per call, so the savings in prompt engineering can be offset by the cost of each individual call taking longer. Fail-safe: Assuming a reasoning model's internal deliberation makes it immune to the same failure modes as any other model, like looping or hallucinating tool parameters, is a mistake; every other guardrail still applies regardless of how much internal reasoning the model does before producing its visible output.