Retrying safely

Some failures are temporary: a busy server, a dropped connection, a rate limit. Trying again later often works. Trying again badly can make things worse: you may hammer a server that is already struggling, or repeat an action twice.

A safe retry follows six rules:

1. Retry only failures that can succeed later: 408, 429, 5xx, connection errors and timeouts.
2. Never retry a non-idempotent action without an idempotency key.
3. Wait between attempts, and wait longer each time. This is called exponential backoff: the wait doubles, for example 0.5, 1, 2, 4 seconds.
4. Add random jitter. If a thousand clients fail at the same moment and all wait exactly 2 seconds, they all come back at the same moment and overload the server again. Randomness spreads them out.
5. Obey Retry-After if the server sent it.
6. Stop after a fixed number of attempts and raise the error. Never retry forever.

Here is the complete function from the example project code/http-client:

RETRYABLE_STATUS = {408, 429, 500, 502, 503, 504}


def backoff_delay(attempt: int, base: float = 0.5, cap: float = 8.0) -> float:
    """Exponential backoff with full jitter."""
    return random.uniform(0, min(cap, base * 2 ** (attempt - 1)))


def get_json(client, url, *, max_attempts=5, sleep=time.sleep, **kwargs):
    for attempt in range(1, max_attempts + 1):
        try:
            response = client.get(url, **kwargs)
        except (httpx.ConnectError, httpx.TimeoutException):
            if attempt == max_attempts:
                raise
            sleep(backoff_delay(attempt))
            continue
        if response.status_code in RETRYABLE_STATUS and attempt < max_attempts:
            sleep(retry_after_seconds(response) or backoff_delay(attempt))
            continue
        response.raise_for_status()   # 4xx fails immediately
        return response.json()

Read it as a story. It tries up to max_attempts times. If the connection fails or times out, it waits and tries again, unless that was the last attempt, in which case it raises the error. If the server answers with a retryable status, it waits, using Retry-After when there is one and the backoff delay otherwise. For any other bad status, raise_for_status() raises an error immediately, which is why a 404 is not retried.

backoff_delay is "full jitter": a random number between 0 and the backoff limit. The limit doubles each attempt (0.5, 1, 2, 4...) and stops growing at the cap of 8 seconds. Real values from one run, for attempts 1 to 6: 0.3, 0.86, 0.44, 3.64, 5.53, 5.4 seconds. They differ on every run, and that is the point.

Notice the sleep parameter. It normally is time.sleep, but a test can pass in a fake that records how long it was asked to sleep instead of waiting (Level 0, module 5). Here is the function against my small test server, which answers 429 twice with Retry-After: 2 and then succeeds:

>>> slept = []
>>> get_json(client, "http://127.0.0.1:8765/flaky", sleep=slept.append)
{'ok': True, 'attempts': 3}
>>> slept
[2.0, 2.0]

Two failures, two waits of exactly the two seconds the server asked for, then success. And a 404 on the same server:

>>> get_json(client, "http://127.0.0.1:8765/nope")
httpx.HTTPStatusError: Client error '404 Not Found' for url 'http://127.0.0.1:8765/nope'

It fails at once, after one request, as the rules say.