Part 6 of 8 · 2 min read
Alerting
From rule to notification.
Alert rules. Rules are written as configuration files, for example in YAML. A rule has a query, a condition, a time it must hold, and labels. A worked example (made up): "fire if CPU is over 90 per cent for 5 minutes". The "for 5 minutes" part stops short spikes from waking someone.
The flow.
- The alert evaluator loads the rules and runs them as queries against the query service, at set intervals.
- If a condition holds long enough, the alert fires.
- The alert manager receives it. It checks rules for filtering, merging and routing. It de-duplicates repeats, groups related alerts into one message, and inhibits low-value alerts when a bigger one is active. For example, if a whole data centre is down, it need not send 400 alerts about each server.
- The alert state is stored in a key-value store, so a restarted manager still knows which alerts are active. The states include inactive, pending, firing and resolved.
- Alerts are put on a queue, so none are lost.
- Consumers send them to email, a paging service, chat or a webhook.
A worked example (made up). A rule checks each minute. Samples are 85, 92, 95, 91, 93, 96, 40 per cent. The condition needs 5 minutes above 90. The second to sixth samples are above 90, five in a row, so the alert fires at the sixth sample. The seventh, 40, resolves it.
Build or buy. Many mature alerting and dashboard tools exist, such as Grafana. A strong interview answer weighs building against buying. Building costs staff time. Buying may restrict custom needs, so consider the company's size and skills.
Visualisation. Dashboards are charts over time. Keep them fast by querying recent data and caching popular graphs.
Common mistake
Alerting on every spike. A rule with no waiting time pages people for problems that fix themselves, and people start to ignore alerts. Page only on what needs a human, as Level 9 explained.
Check yourself: 1. What does the "for 5 minutes" part do? 2. What are de-duplication and grouping? 3. Why store alert state in a key-value store?