Part 6 of 12 · 2 min read
What the simulation measured
Outage policies, rate limiting and bulkheads, each with a reading guide.
How to read the outage table: every row faces the identical 120-second outage; "Primary calls per request" in the outage window is the real cost of each policy, and success is the real benefit.
| Policy (120 s outage, 20 req/s) | Success | Via fallback | Primary calls per request in outage | p95 latency |
|---|---|---|---|---|
| No protection | 68.5% | 0% | 1.00 | 1.0 s |
| Retry x3 | 70.0% | 0% | 4.00 | 1.8 s |
| Retry x3 + budget | 70.0% | 0% | 1.10 | 1.0 s |
| Breaker | 68.3% | 0% | 0.01 | 1.0 s |
| Breaker + fallback | 99.7% | 31% | 0.01 | 1.5 s |
| Retry + budget + breaker + fallback | 99.7% | 30% | 0.01 | 1.5 s |
What to notice: success barely moves until a fallback is added. Retries and the breaker each change what happens to the primary (load, mostly). Only the fallback changes what happens to the user.
How to read the rate-limit and bulkhead table: the first two rows share the same burst and limit. The next two share the same hung tool and worker count. The last is a separate point about layered retries.
| Scenario | Outcome |
|---|---|
| 100-request burst, limit 10/s, no limiter | 90 rejected |
| Same, token bucket | 0 rejected, finishes in 9.5 s |
| Hung tool B, shared pool of 10 | healthy tool A succeeds 3% |
| Hung tool B, separate pools 5 + 5 | healthy tool A succeeds 100% |
| 3 layers x 3 retries | 64 attempts per request |
What to notice: the last row is a warning about combining the earlier sections carelessly. Retrying at three layers (client, gateway, SDK) each with three retries multiplies out to 64 attempts for a single request. That is the same amplification that made the first table's naive "retry x3" row backfire, just worse.
# from code/mini-reliability/reliability.py
def amplification(layers, retries_per_layer):
return (1 + retries_per_layer) ** layers
{{svg:patterns|Rate limiter, timeout, retry with budget, breaker and fallback chain.}} {{svg:results|Outage policies, rate limiting and bulkheads.}}