Part 2 of 12 · 2 min read
Alert on symptoms, not noise: burn rate vs a single threshold
A rule that pages on one noisy minute trains people to ignore pages.
Think of a car alarm that goes off every time a truck drives past, versus one tuned to only go off when someone actually tries the door. The first one technically "works." But everyone on the street has learned to ignore it.
In plain words: an alert rule that fires on any single noisy minute pages constantly on nothing, while one that checks whether the error budget is burning too fast over both a short and a long window catches real problems with far fewer false alarms.
A worked example. Over 7 days of pure noise, a static "error rate above 5% for one minute" rule pages 44 times; a multi-window burn-rate rule pages 2 times on the same noise. On a sudden 40-percent outage, the threshold rule happens to page instantly (minute 0) while burn rate takes 5 minutes — but on a slow 3-percent degradation lasting 4 hours, the threshold rule's apparent 24-minute catch is pure chance (a random noisy minute happened to cross 5%), while burn rate reliably catches it at minute 75.
The key lines. From code/mini-incident/alerting.py:
# from code/mini-incident/alerting.py
def burn_rate_rule(series, slo=0.99, rules=((5, 30, 10.0), (30, 120, 2.0))):
budget = 1 - slo
for t in range(len(series)):
firing = any(window_rate(series, t, s) / budget > f and window_rate(series, t, l) / budget > f
for s, l, f in rules)
Read it as: a page only fires when the error rate, divided by the error budget, is burning too fast in both a short window and a longer one at the same time. A single noisy minute can spike the short window alone. But it cannot also drag the long window over the line. That is exactly what filters it out.
SLI, SLO, error budget, burn rate
An SLI is a measured indicator (like the share of requests that succeed); an SLO is its target (like 99 percent); the error budget is the allowed failure (1 percent here). Burn rate is the observed error rate divided by the error budget — a burn rate of 10 means the budget is being spent ten times too fast.
Example: the 44-to-2 false-page drop above, the direct payoff of checking burn rate instead of a single raw number.