CSE Roadmap

Common tier · about 2 hours · 6 parts

Part 2 of 6 · 1 min read

The shape of an incident

Detect, mitigate, repair, learn.

An incident is an event in which the service is, or is about to be, worse for users than the SLO allows. Picture a fire brigade: first put out the fire, then investigate how it started, then change the building rules. Software incidents follow the same order.

UDS describes the first part, from the point of view of the on-call engineer (module 2):

  1. An alert fires. It links to dashboards and a runbook.
  2. The engineer mitigates: reduces the harm. UDS gives two examples. A new artifact degraded the service, so roll it back. The service cannot cope with load that has not even grown, so scale it out.
  3. Every action goes into a shared channel that other teams can read, so others can help, follow progress and take over.
  4. Only after mitigation does the team ask how to prevent a repeat. The wider the impact, the more time it deserves.
  5. Incidents that burned a large part of an SLO's error budget need a postmortem.

A made-up timeline. 02:10 alert for a burn rate of 14.4. 02:14 on-call acknowledges and sees that a release finished at 02:05. 02:18 rolls back. 02:25 error rate back to normal. Total harm: 15 minutes. The root cause (a missing index in the new query) is found the next day. Fixing the cause at 02:15 while users were failing would have lengthened the damage.

Mitigation

Mitigation is any quick action that reduces user harm, even if it does not fix the underlying fault.

Example: rolling back a release, switching a feature flag off, adding instances, or sending traffic away from a broken region.

Check yourself: 1. Why does mitigation come before finding the root cause? 2. Why are actions written to a shared channel? 3. In the timeline, how long were users harmed?