CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Retry storms: why everyone retrying at once makes it worse

A crowd that all comes back at the same minute, and the one-line fix.

Think of a canceled flight: if the airline tells everyone "boarding resumes in exactly ten minutes," a crowd forms at the gate at minute ten. If it instead tells each passenger a slightly different time within that ten minutes, the same number of people get through without a crush.

In plain words: if every failed client retries after the same fixed delay, their retries all land in the same instant, which can overwhelm the very thing that was just recovering.

A worked example. 1,000 clients all fail at once against a service that can handle 50 requests per second. Without jitter, every client retries after exactly the same delay. The retry attempt is a single spike of 1,000 requests in one second. 9,500 requests get rejected over the whole recovery, and it takes 962 seconds to clear the backlog. With full jitter, each client picks a random delay up to the maximum instead of the maximum itself, spreading retries over a window. The peak drops to 720, rejections to 2,283, and clearing time to 69 seconds.

# from code/mini-queue/jobqueue.py
def backoff(attempt, base=2.0, cap=60.0, jitter="full", rng=None):
    d = min(cap, base * 2 ** min(attempt, 60))
    if jitter == "none": return d
    if jitter == "full": return rng.uniform(0, d)

Read it as: compute the delay that doubles with each failed attempt, capped at a maximum. Then, instead of waiting exactly that long, wait a random amount of time somewhere between zero and that long. Jitter does not remove the very first spike, since every client is still failing for the first time together. It only spreads out what happens after.

Jitter

Randomness added to a retry delay so that clients who failed at the same moment do not all retry at the same moment.

Example: rng.uniform(0, d) instead of returning d itself.