CSE Roadmap

Core tier · about 3 hours · 6 parts

Part 4 of 6 · 2 min read

Bulkheads

Compartments so a leak cannot sink the ship.

A ship's hull is divided into watertight sections. If one floods, the others stay dry. A bulkhead in software does the same for resources.

BM's first bulkhead is the one his team missed: a separate connection pool for each downstream service. If one pool is exhausted, calls to other services still work. He suggests this as the minimum. He also notes that splitting functionality into separate services (Level 7) works as a bulkhead too, since an outage in one area is less likely to reach another.

Bulkhead

A bulkhead splits a shared resource into separate parts so that a problem in one part cannot use up the others.

Example: one pool of 20 connections for each of 4 downstream services, and not one pool of 80 for all of them.

BM says bulkheads are in many ways the most important of the three patterns (timeouts, breakers, bulkheads). Timeouts and breakers free resources once they are constrained. Bulkheads make sure they do not become constrained to begin with.

Bulkheads for clients. UDS applies the idea to users of a service. One greedy or "poisonous" client can send heavy requests that slow every server. A rate limit (page 4) may not help if each request is heavy. So partition the pool of service instances and assign each client to one partition. The book's example: 10 instances in 5 partitions. A bad client can hurt only 1 partition, which is 20 percent of the instances. The unlucky clients in that partition are still fully hit.

Virtual partitions. To improve this, give each client a random subset of instances. UDS's numbers: from a pool of 10 instances, there are 45 ways to choose 2. Each client gets one of these 45 pairs. Two clients rarely share the same pair, and a degraded pair only partly overlaps with others. Add a health check on the load balancer and retries on the client, and the isolation is much better.

A worked example (made up). A bad client is on instances 3 and 7. A client on instances 3 and 8 loses one of its two instances and keeps working at half power while it retries on 8. A client on 1 and 2 is untouched. Under the plain scheme with 5 fixed partitions, every client in the bad partition would lose both its instances.

The cost. UDS warns of two. Too many partitions lose the economy of sharing costly resources among users who are busy at different times. And scaling gets harder: with no partitions you just add instances, but with partitions some are much hotter than others.

Check yourself: 1. Why did one shared pool let the ad service take everything down? 2. In the 10-instance example with 5 partitions, what share of instances can one bad client harm? 3. Why do virtual partitions protect better than fixed partitions?