Part 8 of 8 · 2 min read
Exercises
Easy
Easy 1: Good or bad alert (20 minutes)
Classify five alerts as actionable or not: disk 90 percent, p99 latency above SLO for 10 minutes, memory spike for 1 minute, error budget 5 percent gone in 1 hour, one failed health check.
Done when: a five-row table with a reason each.
Hint
Ask whether the receiver knows the user impact.
Easy 2: Precision and recall (20 minutes)
A month has 5 real outages. An alert fires 20 times, 4 of them on real outages. Compute precision and recall.
Done when: two numbers.
Hint
Divide by alerts for precision, by events for recall.
Easy 3: Burn rate reading (20 minutes)
For a 30-day SLO, give the days to empty the budget at burn rates 1, 2, 6 and 10.
Done when: four numbers with the working.
Hint
Days equal 30 divided by the rate.
Medium
Medium 1: A threshold (30 minutes)
An SLO has a 30-day window. Find the burn rate threshold that fires when 5 percent of the budget is used in a 6-hour window.
Done when: one number with the working, and the hours until empty at that rate.
Hint
Share of window passed is 6 divided by 720.
Medium 2: Two-tier alerts (30 minutes)
Design a page-now alert and an e-mail alert for one SLO using burn rates. State what each means and who acts.
Done when: a table with window, burn rate, action and owner.
Hint
A high rate over a short window pages, a low rate over a long window e-mails.
Medium 3: Fix a dashboard (30 minutes)
A chart shows request count (millions) and p99 latency (milliseconds) on one axis, with 20 lines and no deployment marks. List six fixes.
Done when: six concrete changes.
Hint
Split by range, trim lines, add alert and deploy lines.
Hard
Hard 1: Three dashboards (45 minutes)
For an online store, sketch an SLO dashboard, a public API dashboard and a checkout service dashboard. List 5 to 7 charts for each, in top-to-bottom order.
Done when: three lists with the reason for each chart's place.
Hint
Put the most important charts at the top.
Hard 2: A runbook (45 minutes)
Write a runbook for "p99 latency of the search service above SLO". Include links to look at, mitigation choices and when to escalate.
Done when: a one-page runbook with ordered steps.
Hint
Mitigation steps come before cause-hunting.
Hard 3: Repair a rotation (50 minutes)
A team gets 30 pages a week, and 24 need no action. Propose a plan to bring it down, using precision, burn rates and the stop-features agreement.
Done when: a plan with targets and a review date.
Hint
Count false pages first, then change the worst alerts.