CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 5 of 8 · 2 min read

In the middle of one

Reduce load first, then restore slowly.

UDS says a cascade is very hard to bring under control once it has started. The books do not give a full incident procedure, so this section applies their ideas and marks the rest as common practice.

What follows from the books:

  • Cut the feedback loop. Retries feed the loop. If clients can be told to stop or back off (a Retry-After header, page 4), do so. Open circuit breakers by hand: BM notes that a breaker can be blown manually to make callers fail fast while a service is offline.
  • Shed load. Reject low-priority work so the service can finish what it has (page 4). Drop optional features (graceful degradation).
  • Do not remove a struggling node just because it is slow. DDIA's warning about declaring nodes dead applies. Moving its load to others can finish the job.
  • Do not let the cure flood the patient. In UDS's story, when B comes back it is hit by the entire load and falls over again. A recovering instance should be given traffic in small steps.

Common practice, not from the books used here:

  • Roll back first, investigate later. If a release started the cascade, roll it back.
  • Add capacity only if the bottleneck is capacity. More servers do not help if the cause is a slow dependency or a lock. They may add more load on it.
  • Restore in order. Bring back the dependencies first (database, cache), then the services that use them, then the traffic, in steps such as 10, 25, 50 and 100 percent.
  • Write down the timeline as you go. It feeds the review afterwards.

A worked example (made up). A service has 4 instances at 70 percent load. A slow database makes calls take 10 times longer. Threads fill up, health checks fail and two instances are removed. The two left receive double traffic and fail. Response: (1) turn on shedding of the low-priority endpoints, (2) tell the gateway to cut retries, (3) open the breaker to the database for the non-critical reads, (4) when the database recovers, restore instances one at a time, each taking traffic in 25 percent steps.

Gradual recovery

Gradual recovery means bringing traffic back to a recovered service in small steps, and not all at once.

Example: the load balancer sends 10 percent of normal traffic to a restarted instance, then 25, then 50, then 100 as its error rate stays low.

Check yourself: 1. Why should retries be stopped early in a cascade? 2. Why is it wrong to add servers when the database is the bottleneck? 3. What does gradual recovery protect against?