CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 6 of 8 · 1 min read

Why cascades persist

Some failures keep going after the trigger is gone.

A cascade can keep itself alive after the original cause is fixed. In UDS's story, even when B came back, the pattern repeated. The same happens when a retry storm keeps a service busy: the extra load from retries alone keeps it slow, so more calls time out, so more retries are sent. The trigger is gone and the loop remains.

Ways to break such a loop, drawn from the earlier pages:

  • the breaker opens and sends no calls for a while, so the service gets a quiet period;
  • shedding or rate limiting caps the load below the service's limit;
  • backoff with jitter spreads the retries so the queue can drain;
  • a short, controlled stop of the callers lets the service empty its backlog.

A worked example (made up). A service handles 100 requests a second, and 300 per second arrive because of retries. Each second 200 requests are left unfinished and time out. Each of those is retried, so the next second more arrive. Even if the original cause is fixed, 300 per second persist. Capping the incoming rate at 100 per second for 30 seconds lets the backlog drain, and the loop ends.

Check yourself: 1. Why can the loop continue after the cause is fixed? 2. Which two tools can give a service a quiet period? 3. In the example, what is the arrival rate you must cap below?