Part 4 of 8 · 2 min read
Avalanche: many entries fail together
When a large share of the cache disappears at once, from a shared expiry or a dead server.
Think of an apartment block where every tenant's lease ends on the same day. The office is quiet all year, then swamped on that one day. Spreading the lease dates would smooth the work.
Cache avalanche
A cache avalanche happens when a large part of the cache becomes unavailable at the same time, so that most requests miss and the database receives far more load than usual.
Example: 300,000 entries are filled at midnight with a TTL of 24 hours. At the next midnight they all expire together.
There are two common causes.
Shared expiry. Entries filled together with the same TTL expire together. The fix is jitter (module 3): a random extra time on each TTL.
A worked example. 300,000 entries expire in the same minute. Each is read once a minute on average, so 300,000 reloads happen in that minute, or 5,000 per second. A database that handles 1,000 reads a second is overwhelmed. With jitter spread over 30 minutes, reloads become 300,000 ÷ 1,800 seconds ≈ 167 per second.
Loss of the cache itself. A cache server crashes, or the whole cache is restarted. The book "Building Microservices" warns of this case: if an entire machine or group of machines fails, a large number of requests hit the origin. It notes that origins serving highly cacheable data are often scaled to handle only a fraction of the total traffic, so a thundering herd from a vanished cache region can overwhelm them. Its answer is to hide the origin and refill the cache in the background, failing requests fast.
A worked example of the second cause. A site serves 40,000 reads a second at a 95 percent hit ratio. The database sees 2,000 a second. If the whole cache dies, the database is asked for 40,000 a second, which is 20 times its normal load.
Defences:
- Several cache nodes. The book "System Design Interview" recommends multiple cache servers across data centres, so that no single server is a point of failure. With 10 nodes, the loss of one removes about 10 percent of the entries. At 40,000 reads a second, that node handled 4,000, of which 95 percent (3,800) were hits and now miss. The database goes from 2,000 to 5,800 a second, not to 40,000.
- Load shedding. When the database is at its limit, answer some requests with an error at once (fail fast), so the database can serve the rest.
- Warm-up. Refill the hottest keys before taking traffic after a restart.
- Circuit breaker. Stop calling a failing dependency for a while (a later level returns to this).
Check yourself:
- 600,000 entries expire in a 10-minute window instead of a single minute thanks to jitter. About how many reloads per second?
- Why is a cache spread over several nodes safer than one big cache server?