CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 7 of 8 · 2 min read

Risk management

Fix first what is likely and costly.

Risk management

Not every failure deserves work. The day has limited hours. UDS gives a simple rule: for each failure, multiply the probability that it happens by the impact if it does. The result is a risk score. A failure that is very likely and has a wide impact needs action soon. One that is unlikely and has a small impact can wait.

There are two ways to reduce a risk: lower the probability, or lower the impact.

A worked example (made up). Score each factor from 1 (low) to 5 (high).

Failure Probability Impact Score
One web server crashes 5 1 5
Database primary fails 2 5 10
Certificate expires 3 5 15
Whole region lost 1 5 5

The certificate expiry scores highest, and it is cheap to fix with a reminder or automatic renewal. The web server crash is common but harmless because of the other five servers.

Interview angle

Answer in your head first, then open it.

Why do failures increase as a system grows?

More parts and more operations multiply the chances, and failures are not independent.

What is a cascading failure?

One failure raises the load or risk on other parts, so they fail in turn. Retries and removing servers from a pool often feed it.

How do you find single points of failure?

Ask of each component what happens if it fails, then add redundancy or shrink the blast radius.

How do you decide what to fix?

Score likelihood times impact, and start with the highest.

Check yourself: 1. In the example, why does "one web server crashes" score low in spite of being common? 2. Name the two ways to lower a risk. 3. What is the cheap fix for the highest score in the example?

Words you will meet

Word Meaning
Resilient Keeps doing its job when failures happen
Availability Share of time the system can serve requests
Single point of failure One component whose failure stops the system
Blast radius How much of the system a failure damages
Resource leak A resource taken and never given back
Cascading failure A failure that makes other failures more likely
Feedback loop Output that comes back as input and strengthens itself
Risk score Probability times impact