Part 4 of 8 · 2 min read
Alerting on burn rate
Alert on how fast the budget is going, not on a raw dip.
UDS works through an example. The SLO is 99 percent availability over 30 days. A first idea is to alert when availability falls below 99 percent within a one-hour window. Check what has been spent at that moment. One hour out of 30 days (720 hours) is 1 ÷ 720, about 0.14 percent of the window. Being told that 0.14 percent of the budget is gone is rarely worth a page. The alert catches almost everything (high recall) but is noisy (low precision).
One fix is to require the condition to hold for longer. That raises precision but delays the page during a real outage. UDS prefers a second fix: alert on how fast the budget is being used, the burn rate. It is the share of the budget used, divided by the share of the SLO window that has passed. A burn rate of 1 uses the whole budget in exactly 30 days. A rate of 2 uses it in 15 days, a rate of 3 in 10 days, and so on.
Rearranged, the formula gives a threshold for a chosen loss. UDS's case: alert when 2 percent of the budget is spent inside a one-hour window.
- Budget used: 0.02.
- Share of window passed: 1 hour ÷ 720 hours.
- Burn rate: 0.02 ÷ (1 ÷ 720) = 14.4.
So the alert fires when the burn rate is 14.4 or more, which means the budget is draining 14.4 times faster than a steady spend. At that speed all of it is gone in 720 ÷ 14.4 = 50 hours.
To raise recall without adding noise, use several alerts with different thresholds. UDS suggests tiers. A slow burn, with a rate under 2, can raise a low-severity alert by e-mail that is read in working hours. A fast burn pages a person. UDS points to the Google SRE workbook for detailed examples.
Burn rate
The burn rate is how fast the error budget is being used, as the share of budget spent divided by the share of the window that has passed. A rate of 1 uses it up exactly at the end of the window.
Example: spending 10 percent of the budget in the first 5 percent of the window is a burn rate of 2.
UDS also says some alerts should cover known hard failures you have not yet had time to design or debug away. Its example is a memory leak: an alert on running out of memory can trigger an automatic restart until the real fix is built. This is the white-box monitoring from module 1.
Common mistake
Alerting on the SLI itself over a short window. It fires on tiny dips that cost almost none of the budget, and people learn to ignore it.
Check yourself: 1. What burn rate empties a 30-day budget in 5 days? 2. Why does a longer condition window slow the page? 3. What alert would you send by e-mail rather than phone?