Part 6 of 8 · 2 min read
Steady state
Define normal, so the system can notice when it is not.
Self-healing works only against a clear picture of normal. The steady state is the set of measurements that describe the system when it is doing its job: requests per second, p99 response time, error rate, queue depth, and so on. (The term comes from chaos engineering practice and is not defined in the books used here. UDS's chapter on monitoring supplies the measurements.)
UDS's monitoring chapter says that an alert should be actionable and tied to something users feel. A high CPU alert is not useful on its own, since it is unclear whether it has any impact. A service level objective (SLO) makes a good alert source, since it puts a number on what users feel. UDS also describes error budgets: if the budget is burning too fast or is used up, repair work takes priority over new features.
The loop of a self-healing system:
- Measure against the steady state.
- Detect a departure: a health check fails, a watchdog threshold is crossed, an SLO burns fast.
- Act automatically for known cases: remove from the pool, restart, shed load, open a breaker, fail over.
- Escalate to a human for unknown cases or when automatic actions repeat.
- Learn: review each event and improve the checks.
DDIA gives a caution in its discussion of rebalancing and failure detection: automation combined with automatic failure detection can be dangerous. A human in the loop for some actions is slower, but it can prevent operational surprises. Decide which actions are safe to automate (restart one instance) and which need a person (move a whole database).
Steady state
The steady state of a system is the normal range of its key measurements when it is serving users well.
Example: about 2,000 requests per second, p99 under 400 ms, errors under 0.1 percent, and queues under 50 messages.
Check yourself: 1. Why is "CPU is high" a poor alert? 2. Name two automatic actions and one that should need a human. 3. Why define the steady state before an incident?