Part 2 of 8 · 1 min read
Why failures grow
More parts and more operations mean more failures.
Think of a long string of holiday lights. Each bulb is very reliable, yet the more bulbs on the string, the more likely it is that at least one is dead on any given night.
UDS puts it this way: if an operation has some chance of failing, the total number of failures grows with the number of operations. Scaling out makes it worse. Failures are also not independent. One failing part raises the chance that another fails.
The book gives a leak example. A service loses 1 MB of memory every 100 requests. At 1,000 requests a day it loses 10 MB, and a restart for a new build clears it long before it matters. At 10 million requests a day it loses 100,000 MB, which is 100 GB, within one day. The code is the same. Only the volume changed.
Resilient system
A resilient system keeps doing its job when failures happen.
Example: a shop keeps taking orders while its recommendation service is down.
Resilience protects availability: the share of time the system can serve requests. (Level 0 introduced it.) UDS gives the "nines" table. Two nines (99 percent) allow 14.4 minutes of downtime a day. Three nines (99.9 percent) allow about 1.4 minutes a day, which the book converts to 43 minutes a month. At that size no human can react in time, so the system must heal itself.
Check yourself: 1. In the leak example, why does the same code fail at 10 million requests a day but not at 1,000? 2. How many minutes of downtime a month do three nines allow? 3. Why does independence of failures matter?