CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 6 of 8 · 3 min read

SLOs and error budgets

A target, and the amount of failure it allows.

A service level objective (SLO) is the range of acceptable values for an SLI. Inside it, the service counts as healthy. It also sets users' expectations. UDS's example: 99 percent of calls to an endpoint finish in under 200 ms, measured over a rolling week. Seen the other way, up to 1 percent of requests in the week may be slower. That 1 percent is the error budget: the amount of failure that can be tolerated.

A service level agreement (SLA) is a contract built on an SLO. It says what happens, usually financially, when the SLO is missed.

SLO and error budget

A service level objective is a target for an SLI over a time window. The error budget is the amount of failure the target allows.

Example: an SLO of 99.9 percent of requests succeeding over 30 days gives an error budget of 0.1 percent: 1,000 failed requests out of 1,000,000.

A worked example (made up). The SLO is 99.9 percent availability over 30 days. In minutes, 0.1 percent of 30 days is 0.001 × 43,200 = 43.2 minutes of full outage. A bad release causes 13 minutes of full outage on day 4. It has burned 13 ÷ 43.2 = 30 percent of the budget. UDS says the weight of an incident is the share of the budget it burns: one that used 20 percent calls for more thought than one that used 1 percent.

Using the budget. UDS says SLOs help alerting and help the team balance repair work against features. The team can agree that once the budget is used up, repair items come before new features until the SLO is met again.

Windows. Short windows force quick action on fixes. Long windows suit decisions about which projects to fund. So it makes sense to have several SLOs of different window length.

How strict? If the target is too loose you will miss user-facing problems. If too strict you waste time chasing small gains. Even 100 percent reliability from you does not give users a 100 percent experience, since their last-mile connection is outside your control. UDS's advice:

  • Start with comfortable targets and tighten them as confidence grows.
  • Do not just copy what the service meets today, which may be out of reach after load grows. Work back from what users care about.
  • Above three nines of availability is very costly and gives diminishing returns.
  • Keep SLOs few, documented and reviewed. If an issue creates many support tickets and no SLO moved, the SLOs are too relaxed or you are not measuring the right thing.
  • Agree on them with several people. Engineers must agree the target is reachable without too much toil. Product managers must agree it gives a good user experience. UDS quotes Google's SRE book: if you can never win an argument about priorities by quoting an SLO, it is probably not worth having.

Do not over-deliver. Users can start to rely on the real behaviour and not the published SLO. UDS suggests injecting controlled failures in production (chaos testing, Level 8) to check that dependents cope with the stated service level.

Common mistake

Setting the SLO at the level the service happens to meet today, or at 100 percent. The first hides growing risk, and the second leaves no budget for any change.

Check yourself: 1. How many minutes of outage does 99.99 percent allow in 30 days? 2. What should happen to feature work when the budget is gone? 3. Why have more than one window?