Part 7 of 8 · 2 min read
Risk management
Fix first what is likely and costly.
Risk management
Not every failure deserves work. The day has limited hours. UDS gives a simple rule: for each failure, multiply the probability that it happens by the impact if it does. The result is a risk score. A failure that is very likely and has a wide impact needs action soon. One that is unlikely and has a small impact can wait.
There are two ways to reduce a risk: lower the probability, or lower the impact.
A worked example (made up). Score each factor from 1 (low) to 5 (high).
| Failure | Probability | Impact | Score |
|---|---|---|---|
| One web server crashes | 5 | 1 | 5 |
| Database primary fails | 2 | 5 | 10 |
| Certificate expires | 3 | 5 | 15 |
| Whole region lost | 1 | 5 | 5 |
The certificate expiry scores highest, and it is cheap to fix with a reminder or automatic renewal. The web server crash is common but harmless because of the other five servers.
Interview angle
Answer in your head first, then open it.
Why do failures increase as a system grows?
More parts and more operations multiply the chances, and failures are not independent.
What is a cascading failure?
One failure raises the load or risk on other parts, so they fail in turn. Retries and removing servers from a pool often feed it.
How do you find single points of failure?
Ask of each component what happens if it fails, then add redundancy or shrink the blast radius.
How do you decide what to fix?
Score likelihood times impact, and start with the highest.
Check yourself: 1. In the example, why does "one web server crashes" score low in spite of being common? 2. Name the two ways to lower a risk. 3. What is the cheap fix for the highest score in the example?
Words you will meet
| Word | Meaning |
|---|---|
| Resilient | Keeps doing its job when failures happen |
| Availability | Share of time the system can serve requests |
| Single point of failure | One component whose failure stops the system |
| Blast radius | How much of the system a failure damages |
| Resource leak | A resource taken and never given back |
| Cascading failure | A failure that makes other failures more likely |
| Feedback loop | Output that comes back as input and strengthens itself |
| Risk score | Probability times impact |