CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 3 of 8 · 2 min read

The amplifiers

Six ways one failure becomes many.

Amplifier How it spreads the failure Where covered
Load moves to the survivors The failed server's share goes to the others, which then exceed their limits Page 1
Retries Clients send the failed requests again, adding load Page 2
Retries at several levels The load multiplies down the chain Page 2
Resource exhaustion by waiting Threads and connections held by slow calls run out Pages 1 and 3
Automatic failure detection A slow node is declared dead, so its load moves DDIA, below
Automatic rebalancing Data moves to relieve a node but adds work to an already busy cluster DDIA, below

DDIA shows the last two. A node under load may answer slowly. Other nodes decide it is dead. Declaring a node dead means its work is moved to the others, which adds load to them and to the network. In the extreme, DDIA writes, all nodes declare each other dead and everything stops working. Rebalancing is similar. If one overloaded node is slow, other nodes may conclude it is dead and automatically rebalance the cluster to move load off it. That puts more load on the slow node, the others and the network, which worsens the situation. For that reason DDIA suggests that a person may approve the move before data is rebalanced. It is slower, but it can prevent surprises.

DDIA adds another danger of declaring a node dead too early: a node that was alive and part-way through an action, such as sending an email, may find that a second node has taken over the job, so the action runs twice.

A worked example (made up). Five servers each handle 160 requests per second out of a limit of 200. Total load is 800. One fails and its share goes to the other four: 200 each, exactly at the limit. A second fails and the three left must take 266 each, which is over the limit. They slow down, are declared dead by the health check, and the last servers carry everything. A system with 20 percent headroom survived one failure and not the second.

Check yourself: 1. In the example, how much headroom would let the system survive two failures? 2. Why does DDIA prefer a human in the loop for rebalancing? 3. Why can a slow node be worse than a dead one for a failure detector?