CSE Roadmap

Common tier · about 3 hours · 12 parts

Part 4 of 12 · 2 min read

Severity and the incident clocks

What decides who wakes up, and the timestamps that measure how well the response actually went.

Think of a hospital triage desk. A patient is sorted into a category within seconds of arriving, before anyone has diagnosed exactly what is wrong. The purpose is only to decide how fast they need attention.

In plain words: severity is a quick classification based on visible impact (money, data, how many tenants) that decides response speed, and once the incident is underway, a handful of clean timestamps tell you — after the fact — how well that response actually went.

A worked example. An incident with data exposed, or more than $10,000 moved, is automatically SEV1, regardless of anything else about it. One with under 90 percent availability, more than a quarter of tenants affected, or any money moved at all, is SEV2. Once an incident is underway, four timestamps (started, detected, acknowledged, mitigated, resolved) produce the clocks that matter: time to detect, time to acknowledge, time to mitigate, time to resolve — the numbers a team should be trying to shrink release over release.

The key lines.

# from code/mini-incident/incident.py
def severity(a):
    if a.get("data_exposed") or a.get("money_moved", 0) > 10000:
        return "SEV1"
    share = a.get("tenants_affected", 0) / max(1, a.get("tenants_total", 1))
    if a.get("availability_pct", 100) < 90 or share > 0.25 or a.get("money_moved", 0) > 0:
        return "SEV2"

Read it as: each rule checks one visible fact about the incident and returns as soon as one matches. Severity is decided from what is already known in the first minute. It is not decided from a root cause that might take hours to find.

MTTD, MTTA, MTTM, MTTR

Mean time to detect, acknowledge, mitigate and resolve — each measured from the start of an incident to that event, averaged over incidents, and tracked as the numbers an incident process is trying to improve.

Example: the metrics() call above, computing each of these four gaps directly from the incident's own timestamps.