Part 2 of 8 · 1 min read
A useful alert
The person who answers should know at once why it matters.
An alarm in a house is useful when it tells you which room is on fire. One that rings for every passing truck is soon ignored. An alert is the piece of a monitoring system that starts an action once a condition holds, such as a metric passing a threshold. The action can be automatic, such as restarting a service instance, or it can ring the phone of a person who is on call. This page mostly covers the second kind.
The SRE book gives a short list of what to watch first, the four golden signals: latency, traffic, errors and saturation. If a page measures all four, and calls a person only when one of them looks wrong, it covers most user-facing trouble.
UDS says a useful alert must be actionable. The operator should not have to dig through dashboards to learn how bad it is or how urgent. A spike in CPU use fails this test: it is not clear whether any user is affected. An SLO passes it, because an SLO already says how much users are hurt. The error budget can be watched, and an alert can fire when a large share of it is gone.
Actionable alert
An actionable alert states an impact on users clearly enough that the person who receives it knows whether to act now and where to start.
Example: "Checkout has used 2 percent of its monthly error budget in the last hour" is actionable. "CPU above 80 percent on host 14" is not.
Check yourself: 1. What two kinds of action can an alert trigger? 2. Why is a CPU spike a weak thing to page someone for? 3. Why is an SLO a good basis for an alert?