CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 2 of 12 · 2 min read

Why retrying a dead service makes things worse

Redialing a disconnected number faster does not connect the call.

Think of redialing a phone number that is simply disconnected. Dialing it three times instead of once does not raise your odds of getting through; it just ties up your own phone for three calls instead of one.

In plain words: when a dependency is fully down, retrying does almost nothing for success but multiplies the load being sent to a service that is already failing.

A worked example. During a 120-second outage, with no protection at all, 68.5 percent of requests still succeed (the outage is partial — some baseline failures outside it too). Adding three retries barely moves success to 70.0 percent, but it now sends 4.0 calls per request to a primary that is down. Adding a retry budget — capping retries to roughly a tenth of normal traffic — keeps that same 70.0 percent success while cutting load back to 1.10 calls per request.

The key lines. From code/mini-reliability/reliability.py:

# from code/mini-reliability/reliability.py
class RetryBudget:
    def __init__(self, ratio=0.1, cap=10.0): self.ratio, self.cap, self.tokens = ratio, cap, cap
    def on_request(self): self.tokens = min(self.cap, self.tokens + self.ratio)
    def try_retry(self):
        if self.tokens >= 1 - 1e-9: self.tokens -= 1; return True
        return False

Read it as: every request earns a small fraction of a retry token, and a retry can only be spent if enough tokens have built up. Retries stay a small multiple of traffic no matter how many requests are failing at once. A timeout on every call and a deadline on the whole run make any of this possible. Without them, a hung call just holds a worker forever. It does not fail fast enough to retry or give up.

Timeout, deadline, retry budget

A timeout is the longest a single call may take before being abandoned; a deadline is the absolute time the whole request must finish by. A retry budget caps retries to a fraction of normal traffic so they cannot pile onto an outage.

Example: the 1.10-calls-per-request result above, instead of 4.0, because the budget ran out of tokens.