Part 2 of 6 · 2 min read
Latency kills
A slow dependency took down a whole system.
BM tells a story from his own work. A system called a downstream legacy ad service. That service began to respond slowly and then to return errors. The calling system used an HTTP connection pool, and its threads had timeouts for the downstream call, which was good. The trouble was that every thread now spent a long time waiting for the timeout. New requests asked the pool for a worker. None was free, so they hung too. The pool library had a timeout for waiting on a worker, but it was switched off by default.
The result: the application normally held 40 concurrent connections. In five minutes it climbed to about 800 and the system fell over. The ad feature was used by fewer than 5 percent of customers and earned even less. A minor feature took down everything.
BM's conclusion is that a service which answers slowly causes more trouble than one which fails at once. In a distributed system, latency kills.
Nygard names the same idea Fail Fast. If a system can tell in advance that it will fail, it should say so at once. His example is a load balancer with no working server behind it. It should refuse the connection now, not hold it in a queue and hope. His bulkheads pattern splits capacity into separate pools, like the sealed rooms of a ship. An airline gave check-in its own servers, so a flood of flight-status lookups in bad weather could not stop passengers from checking in. He listed three faults in the design:
- The timeouts were not right.
- One shared connection pool served every outbound call, so one slow service could use up all the workers.
- The caller kept sending traffic to a service it knew was unhealthy, which gave the service no chance to recover.
The fixes: set timeouts properly, add bulkheads to separate the pools, and add a circuit breaker to stop sending calls to an unhealthy system. The next sections cover the last two.
Fail fast
Fail fast means to report an error at once instead of waiting and trying a call that is likely to fail.
Example: when the ad service is known to be down, the page skips the ad box in a few milliseconds and does not wait five seconds.
Check yourself: 1. Why did the pool run out of workers when the downstream service was only slow? 2. Why does BM prefer a dead dependency to a slow one? 3. Name the three fixes.