Part 2 of 8 · 2 min read
Monitoring to observability
Symptoms are found with metrics; causes are found with events.
A doctor's thermometer shows a fever but not the illness. UDS makes the same point. A distributed system is never fully healthy, since something is always failing somewhere. Retries, circuit breakers (Level 8) and relaxed consistency let it tolerate that, but they add complexity and make odd behaviour harder to predict. Not every unhealthy state can be watched in advance, so people need tools to debug failures nobody foresaw.
Debugging means forming a guess and testing it. UDS's example: an operator sees the spread of response times growing slowly for weeks, links it to rising traffic, and guesses the service is nearing a limit or a contended resource. Charts alone cannot confirm that guess.
Observability is a set of tools that give detailed insight into a running system, so that unexpected behaviour can be understood. A good platform keeps the time to test a guess short. That needs detailed events with rich context, since nobody knows beforehand which detail will matter.
The sources of data are metrics, event logs and traces. UDS notes the trade-off: - Metrics sit in time-series stores that take high volume but cope poorly with many dimensions. - Logs and traces sit in stores that cope well with many dimensions but poorly with high volume. - Metrics mainly serve monitoring. Logs and traces mainly serve debugging.
Observability is a superset of monitoring. Monitoring finds symptoms. Observability also helps explain the cause.
Observability
Observability is the ability to understand what happens inside a running system from the data it emits: metrics, logs and traces.
Example: an alert says checkout is slow (monitoring). Reading the request traces shows that one database call takes 4 seconds (observability).
Check yourself: 1. Why can monitoring not cover every failure in advance? 2. Which of the three sources handle many dimensions well? 3. Which of them mainly serve monitoring?