CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 6 of 12 · 2 min read

What the simulation measured

Outage policies, rate limiting and bulkheads, each with a reading guide.

How to read the outage table: every row faces the identical 120-second outage; "Primary calls per request" in the outage window is the real cost of each policy, and success is the real benefit.

Policy (120 s outage, 20 req/s) Success Via fallback Primary calls per request in outage p95 latency
No protection 68.5% 0% 1.00 1.0 s
Retry x3 70.0% 0% 4.00 1.8 s
Retry x3 + budget 70.0% 0% 1.10 1.0 s
Breaker 68.3% 0% 0.01 1.0 s
Breaker + fallback 99.7% 31% 0.01 1.5 s
Retry + budget + breaker + fallback 99.7% 30% 0.01 1.5 s

What to notice: success barely moves until a fallback is added. Retries and the breaker each change what happens to the primary (load, mostly). Only the fallback changes what happens to the user.

How to read the rate-limit and bulkhead table: the first two rows share the same burst and limit. The next two share the same hung tool and worker count. The last is a separate point about layered retries.

Scenario Outcome
100-request burst, limit 10/s, no limiter 90 rejected
Same, token bucket 0 rejected, finishes in 9.5 s
Hung tool B, shared pool of 10 healthy tool A succeeds 3%
Hung tool B, separate pools 5 + 5 healthy tool A succeeds 100%
3 layers x 3 retries 64 attempts per request

What to notice: the last row is a warning about combining the earlier sections carelessly. Retrying at three layers (client, gateway, SDK) each with three retries multiplies out to 64 attempts for a single request. That is the same amplification that made the first table's naive "retry x3" row backfire, just worse.

# from code/mini-reliability/reliability.py
def amplification(layers, retries_per_layer):
    return (1 + retries_per_layer) ** layers

{{svg:patterns|Rate limiter, timeout, retry with budget, breaker and fallback chain.}} {{svg:results|Outage policies, rate limiting and bulkheads.}}