Part 9 of 12 · 2 min read
Mistakes that turn a blip into an outage
Common mistakes, and what the example does not do.
Mistakes that turn a blip into an outage
- No timeouts, so hung calls hold workers.
- Retrying at every layer and on non-idempotent actions.
- No jitter or budget, causing retry storms.
- A breaker with no fallback, so users just see fast errors.
- Untested fallbacks, or fallbacks that violate data or quality rules.
- Silent degradation with no signal to the user or dashboards.
- One shared worker pool for all tools and tenants.
- Counting only provider errors and ignoring tail latency.
- Treating availability patterns as a fix for wrong answers.
What the example does not do
About this example What the example does not do
- Clean outage model: the primary fails fast and totally; real failures include slowness, partial errors, brownouts and bad answers.
- Independent fallback: real fallbacks may share a region, a network or a bug.
- Fixed rates and constants: failure rate, breaker thresholds, backoff and capacities are assumptions, not tuned values.
- No concurrency or queues: the time-stepped loop ignores in-flight calls, timeouts and server-side queues.
- Bulkhead model is a toy (fixed durations, rejection instead of queuing).
- No shedding or priority, no hedged requests, no adaptive concurrency.
- Optional real experiment: point your agent at a proxy you can make fail (fault injection, a mock server with 100 percent errors, delays and 429s) and measure success, calls to the failing service and latency with each layer on and off.
Check yourself
A provider is fully down and every client retries three times. Does that help?
Show answer
No. Retry x3 raised success only from 68.5 to 70.0 percent while sending 4.0 calls per request to a service that was already down. Use a retry budget, a circuit breaker and a fallback.
For the curious Retries multiply across layers
Three retries at each of three layers (client, gateway, SDK) make 64 attempts for one user request. Retry at one layer only, with a budget, honouring Retry-After and never retrying non-idempotent actions without keys. A shared pool let a hung tool take every worker, so the healthy tool succeeded 3 percent of the time; separate pools gave 100 percent. All failure rates and durations are assumed.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Set a timeout on every call | And a shorter deadline than your caller. |
| Retry once, with backoff, jitter and a budget | Not at every layer. |
| Use a circuit breaker | It protects the dependency and your latency. |
| Test the fallback regularly | An untested fallback fails during the outage. |