Part 3 of 8 · 2 min read
Failure domains
Spread the copies across things that fail separately.
Redundancy only works if the copies do not fail together. Three servers in one rack are all lost if the rack's power fails. A failure domain is a set of things that can fail at the same time for one cause.
| Domain | Typical cause of failure | Fix |
|---|---|---|
| Machine | Disk, memory, power supply | Several machines |
| Rack | Rack power or network switch | Spread across racks |
| Zone or data centre | Power, cooling, network, fire | Spread across zones |
| Region | Major disaster, regional outage, bad region-wide deployment | Spread across regions |
| Provider | Company-wide failure, contract or account problem | A second provider |
BM covers this in its scaling chapter. Not all services should run in one rack. If a storage area network (SAN) holds all the virtual disks and it fails, it takes all the connected virtual machines. He has seen big, expensive SANs designed not to fail do exactly that, at least twice in ten years.
AWS is split into regions, which BM says to think of as distinct clouds, and each region into two or more availability zones (AZs), roughly its data centres. AWS gives no guarantee for a single node or even a whole zone. For its compute service it offers only 99.95 percent over a month for the region as a whole. So workloads must be spread across zones. Some people also go across regions.
A service level agreement (SLA) matters here. BM's example: if you need to be down no more than four hours a quarter and your host guarantees only eight, either the target or the design must change. Providers also limit their liability, so a missed target may cost you customers while you have little legal recourse. BM advises working out what happens if a supplier fails, and some clients keep a disaster recovery platform with a different supplier.
A worked example (made up). A shop runs 6 servers in zone A. A zone outage takes it down. Moving to 3 servers in each of zones A and B looks the same, but one zone outage now leaves 3 servers, half the capacity. If each server runs at 40 percent, the 3 left run at 80 percent. That is tight but alive. Three zones with 2 servers each would leave 4 servers at 60 percent. More zones lowers the extra capacity needed: with N zones, the survivors must carry the full load, so each server can run at most at (N−1)/N of its safe limit in normal times.
Check yourself: 1. Why does a spread across racks not protect against a zone outage? 2. In the example, what is the highest normal load per server with 3 zones? 3. Why does BM suggest knowing the supplier's SLA?