Part 4 of 13 · 2 min read
Serial stages multiply availability down, and retries only fix the independent kind of failure
Retrying a failed call helps when the failure was a one-off blip, and does nothing when the whole provider is down.
Think of a chain of locks on a canal. If each lock works 99% of the time independently, a boat passing through ten of them in a row has a meaningfully worse chance of a smooth trip than passing through just one. But if a single drought shuts every lock on the canal at once, retrying at one lock does nothing.
In plain words: availability is the share of requests that finish successfully. Serial stages multiply their individual availabilities down (two stages at 99% give about 98%, not 99%). Retries raise availability by giving a stage a second independent chance to succeed. But a correlated failure, where one cause like a provider outage breaks many stages together, makes retries far less useful, because the retry shares the same broken cause as the original attempt.
A worked example. With zero retries allowed on any LLM call, the full design's simulated availability is 98.61%. Allowing one retry per call raises that to 99.91%, a meaningful jump, because most of the simulated failures are modeled as independent blips a second attempt can clear. Allowing two or three retries gains essentially nothing more (still 99.91%), because the model has already captured the independent-failure risk. The model doesn't capture a correlated outage at all, so this 99.91% should be read as an upper bound, not a guarantee.
The key lines. From code/mini-sysdesign/budget.py:
# from code/mini-sysdesign/budget.py
def retry_availability(avail, retries):
"""Chance a stage eventually succeeds if attempts are independent."""
return 1 - (1 - avail) ** (retries + 1)
Read it as: the formula's entire structure depends on "if attempts are independent," stated right in the docstring. (1 - avail) ** (retries + 1) is the chance every attempt fails. That calculation is only valid when each attempt's failure is unrelated to the others', which is precisely the assumption a real provider outage violates.
Availability, correlated failure
Availability is the share of requests that finish successfully; serial stages multiply, so two stages at 99% give about 98%. A correlated failure is one cause (a provider outage) breaking many stages together, which makes independent-failure arithmetic overstate real availability.
Example: the retry curve flattening out at 99.91% after just one retry is the independent-failure ceiling. A real provider outage would pull the actual number below that ceiling, in a way this simulation, by its own stated assumptions, cannot show.