CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 7 of 8 · 3 min read

On-call

A person answers the alert, so the system must be built for that person.

On-call

On-call means a named engineer is ready to respond to alerts outside normal hours. UDS says a healthy rotation is possible only if services are built from the start with reliability and ease of operation in mind. When the developers who build a service also operate it, they have a reason to cut the operating burden, and they know the design best.

UDS stresses care for the people: - Being on call is stressful, even without call-outs, because of the lost freedom. It should be paid. - The on-call engineer should not be expected to make progress on feature work. They are interrupted anyway, so they get free rein to improve dashboards, runbooks and resilience. - Alerts must be actionable. Each should link to relevant dashboards and a runbook, a written list of steps, because it is easy to miss a step in the middle of the night. UDS also says to automate what can be automated: machines follow instructions well. - Actions taken should be posted in a shared channel, such as a team chat that other teams can read. Others can chime in, follow progress, and take over.

Mitigate first. The first step is to stop the harm, not to find the root cause. A bad release? Roll it back. Load too high for the capacity? Scale out. After that, think about how to prevent a repeat, spending more time when the impact was wider. Incidents that burned a significant part of the error budget need a postmortem, covered in module 5.

The team agreement. UDS says the team should agree in advance: when the budget is spent or alerts run wild, new feature work stops and everyone fixes reliability until the on-call load is healthy.

Runbook

A runbook is a short written guide to the steps an operator should take when a particular alert fires.

Example: "Queue depth high: 1) check the consumer dashboard, 2) if consumers are down, restart them, 3) if not, scale to 10 consumers."

Interview angle

Answer in your head first, then open it.

What makes a good alert?

It is actionable, tied to user impact (an SLO), links to a dashboard and a runbook, and does not fire for harmless dips.

Why alert on burn rate?

A raw threshold over a short window is noisy. A burn rate alert fires only when the budget drains fast enough to matter, and still fires quickly.

First action when paged?

Mitigate: roll back or scale out. Fix the root cause afterwards.

What if the team gets too many alerts?

Stop feature work and restore a healthy rotation.

Check yourself: 1. Why should developers be on call for their own service? 2. What are the two quick mitigations UDS names? 3. What should happen to feature work if alerts spiral?

Words you will meet

Word Meaning
Alert An action triggered when a monitored condition is met
Actionable Clear enough about impact that the receiver knows what to do
Precision Share of alerts that point to real events
Recall Share of real events that raised an alert
Burn rate Speed at which the error budget is being used
Dashboard A screen of charts for one audience
Runbook Written steps for responding to an alert
On-call Being ready to respond to alerts outside normal hours
Mitigate Reduce the harm before finding the cause