Covers: Definitions; The layers; Results; Design rules; Interview questions, with the follow-ups; Common mistakes; What the example does not do.
Definitions
Timeout, deadline
Timeout: the longest a call may take before it is abandoned.
Deadline: an absolute time by which the whole request must finish, passed down to each call.
Retry budget, backoff, jitter
Retry budget: a cap on retries as a fraction of normal traffic.
Backoff and jitter: increasing, randomised waits between retries (module 2).
Circuit breaker
Circuit breaker: a switch that stops calls to a failing dependency and tests it periodically.
Half-open: the state where a single probe call decides whether to resume.
Fallback, graceful degradation
Fallback: an alternative when the primary fails.
Graceful degradation: reduced but useful service instead of failure.
Bulkhead, load shedding
Bulkhead: separate resource pools so one failure cannot consume everything.
Load shedding: deliberately rejecting some work to keep the rest healthy.
Token bucket
Token bucket: a rate limiter that allows bursts up to a capacity and refills at a steady rate.
The layers
Rate limiter, timeout, retry with budget, breaker and fallback chain.
Results
Policy (120 s outage, 20 req/s)
Success
Via fallback
Primary calls per request in outage
p95 latency
No protection
68.5%
0%
1.00
1.0 s
Retry x3
70.0%
0%
4.00
1.8 s
Retry x3 + budget
70.0%
0%
1.10
1.0 s
Breaker
68.3%
0%
0.01
1.0 s
Breaker + fallback
99.7%
31%
0.01
1.5 s
Retry + budget + breaker + fallback
99.7%
30%
0.01
1.5 s
Scenario
Outcome
100-request burst, limit 10/s, no limiter
90 rejected
Same, token bucket
0 rejected, finishes in 9.5 s
Hung tool B, shared pool of 10
healthy tool A succeeds 3%
Hung tool B, separate pools 5 + 5
healthy tool A succeeds 100%
3 layers x 3 retries
64 attempts per request
Outage policies, rate limiting and bulkheads.
Retries alone do not fix an outage; they multiply load on a service that is down.
A breaker protects the dependency; a fallback protects the user. You want both.
Smoothing beats bouncing for rate limits.
Isolation beats capacity: more workers would not have saved tool A.
Design rules
Put timeouts on every call and a deadline on every run; keep inner timeouts shorter than outer ones.
Retry only transient errors, at one layer, with backoff, jitter and a budget; honour Retry-After; only retry idempotent or keyed operations.
Use a circuit breaker per dependency (per provider, per tool) with a probe; alert when it opens.
Define a fallback chain and test it: secondary provider or model, cached answer, degraded tools, human handoff; log which path served each request.
Check fallback permissions (data residency, regulation, quality floor) before using them (Level 10 module 10).
Rate-limit on the client with a shared token bucket; queue background work, shed interactive work.
Isolate with bulkheads: per-tool and per-tenant pools or concurrency limits (module 7).
Degrade honestly: tell users when an answer is reduced; do not silently return stale or weaker output for critical tasks.
Run failure drills: inject outages, slowness and 429s in staging and watch the dashboards (module 11).
Alert on symptoms (error rate, p95, breaker state, fallback rate) not just causes.
Interview questions, with the follow-ups
Your model provider goes down. What happens?
"Calls fail fast under a timeout, a retry budget stops retries from amplifying load, the breaker opens so we stop calling the failing provider, and the gateway falls back to a second provider or a smaller model, with the user told if the answer is degraded. In my simulation that took success from 68 to 99.7 percent during a 120-second outage." Follow-up: "Why not just retry?" Retries alone left success at 70 percent and sent 4 calls per request to a dead service.
Explain the circuit breaker.
"Closed: calls flow and I track failures. If the failure rate over the last window crosses the threshold it opens and calls fail immediately for a cooldown. Then half-open: one probe; success closes it, failure reopens it. It stops hammering a failing dependency and gives callers fast failure." Follow-up: "What do you tune?" Window, threshold and cooldown, trading false opens against slow detection.
What is retry amplification?
"Retries multiply across layers: three retries at each of three layers is 64 attempts per request, which can turn a blip into an overload. I retry at one layer, cap retries to roughly a tenth of traffic with a budget, and use backoff with jitter." Follow-up: "Which errors do you retry?" Timeouts, 5xx and 429 with Retry-After; not 4xx validation errors, and not non-idempotent actions without an idempotency key.
Why bulkheads?
"One hung dependency can hold every shared worker. In my test a shared pool let a hung tool drop the healthy tool's success to 3 percent; separate pools kept it at 100 percent." Follow-up: "Per tool or per tenant?" Both can matter: per tool for dependency failures, per tenant for noisy neighbours (module 7).
Common mistakes
No timeouts, so hung calls hold workers.
Retrying at every layer and on non-idempotent actions.
No jitter or budget, causing retry storms.
A breaker with no fallback, so users just see fast errors.
Untested fallbacks, or fallbacks that violate data or quality rules.
Silent degradation with no signal to the user or dashboards.
One shared worker pool for all tools and tenants.
Counting only provider errors and ignoring tail latency.
Treating availability patterns as a fix for wrong answers.
What the example does not do
Clean outage model: the primary fails fast and totally; real failures include slowness, partial errors, brownouts and bad answers.
Independent fallback: real fallbacks may share a region, a network or a bug.
Fixed rates and constants: failure rate, breaker thresholds, backoff and capacities are assumptions, not tuned values.
No concurrency or queues: the time-stepped loop ignores in-flight calls, timeouts and server-side queues.
Bulkhead model is a toy (fixed durations, rejection instead of queuing).
No shedding or priority, no hedged requests, no adaptive concurrency.
Optional real experiment (I did not run this): point your agent at a proxy you can make fail (fault injection, a mock server with 100 percent errors, delays and 429s) and measure success, calls to the failing service and latency with each layer on and off.
Check yourself
A provider is fully down and every client retries three times. Does that help? (No. Retry x3 raised success only from 68.5 to 70.0 percent while sending 4.0 calls per request to a service that was already down. Use a retry budget, a circuit breaker and a fallback.)
For the curious Retries multiply across layers
Three retries at each of three layers (client, gateway, SDK) make 64 attempts for one user request. Retry at one layer only, with a budget, honouring Retry-After and never retrying non-idempotent actions without keys. A shared pool let a hung tool take every worker, so the healthy tool succeeded 3 percent of the time; separate pools gave 100 percent. All failure rates and durations are assumed.
What you will use daily at work
Practice
What it looks like
Set a timeout on every call
And a shorter deadline than your caller.
Retry once, with backoff, jitter and a budget
Not at every layer.
Use a circuit breaker
It protects the dependency and your latency.
Test the fallback regularly
An untested fallback fails during the outage.
Draft: confirm your model gateway and SDK default retry and timeout settings so you do not stack retries.
What you must be able to say (one page)
Every dependency of an agent will fail: model APIs, tools, vector stores. An agent with no retry or fallback logic can crash outright when a single API call fails, leaving the user waiting and confused. Design tools for graceful failure, watch rate limits, validate outputs, retry strategically, fall back gracefully and log every step. A model gateway can limit usage and apply fallback policies to ride out rate limits or provider failures, which are common.
The layers, outermost to innermost: client-side rate limiter, per-call timeout, retry with backoff and jitter under a retry budget, circuit breaker, fallback chain. Add bulkheads (separate pools) and load shedding. Each solves a different failure.
Simulation (provider fully down for 120 s of a 400 s run, 20 requests per second, 2 percent baseline errors; assumed behaviour): with no protection, 68.5 percent of requests succeeded and the primary received 1 call per request. Retry x3 barely helped (70.0 percent) while sending 4.0 calls per request to a service that was already down. A retry budget (retries capped near 10 percent of traffic) kept the same success with 1.10 calls per request. A circuit breaker cut calls to the failing primary to 0.01 per request (just probes) but, alone, did not restore success (68.3 percent). Breaker plus fallback (a second provider, assumed 99 percent available) restored 99.7 percent success, with 31 percent of requests served by the fallback and p95 latency of 1.5 s.
Retries amplify. Three retries at each of three layers (client, gateway, SDK) is 64 attempts for one user request. Retry at one layer only; use a budget; honour Retry-After; never retry non-idempotent actions without idempotency keys (modules 1 to 3).
Circuit breaker states: closed (calls flow) -> open (failure rate over the last 20 calls at least 50 percent: block calls for 10 s) -> half-open (allow one probe) -> closed on success or open again on failure. The breaker protects the dependency (it stops hammering it) and your latency (fail fast). Level 10 module 9 used the same idea as a safety kill switch; here it protects availability.
Fallbacks need design. Options: another model or provider (the gateway's job), a smaller model, a cached or stale answer, a degraded mode with fewer tools, a canned "try later", or a human handoff. Decide which are allowed (regulated tasks may forbid a different model), tell the user the answer is degraded, and test them regularly: an untested fallback is the one that fails during the outage.
Rate limits: smooth, do not bounce. A burst of 100 requests against a provider allowing 10 per second got 90 rejected (429) with no limiter; a token bucket (burst 5) rejected 0 and finished in 9.5 s. Share the limiter across workers (module 2) and prefer queued delay for background jobs, rejection or shedding for interactive ones.
Bulkheads isolate failures. With 10 workers shared between a healthy tool A and a tool B that hangs for 30 s, A succeeded only 3 percent of the time (B's hung calls held every worker); with separate pools (5 + 5) A succeeded 100 percent. Combine with timeouts so hung calls release workers.
Timeouts everywhere. A call without a deadline holds a worker forever; set timeouts per call and per run (module 5 budgets), shorter than the caller's timeout so the caller sees your error, not its own.
Honest limits: simulated provider with assumed failure modes (clean outage, fast failures, independent fallback), no latency distributions or partial degradations, and no correlated failures between primary and fallback.