Part 7 of 8 · 2 min read
Putting it together
Metrics and traces are summaries made from events.
How the three fit together
UDS's last point: event logs are fine-grained and specific to one service, so one event cannot show a whole request or the health of a service. Metrics and traces are derived views built from events and tuned for specific uses.
- A metric is a time series of summary numbers, made by combining counters or observations over many events. Some systems have services emit counters in events and let the back end roll them up.
- A trace can be built by gathering all events from the life of one request into an ordered list. Services can emit span events and let the back end assemble them.
A typical debugging path: an alert fires (module 2), the service dashboard shows which endpoint is slow, a trace shows which step is slow, and the log event of that step shows the cause.
Interview angle
Answer in your head first, then open it.
What is the difference between monitoring and observability?
Monitoring finds symptoms. Observability gives the data to explain them.
How do you debug one slow request across services?
Follow its trace id through the spans, then read the log events of the slow span.
What belongs in a log event?
Who, what, success or failure, timings, status of each call, request id, and no sensitive data.
How do you control logging cost?
Levels, sampling with more weight on failures, collector rate limits, and in-memory aggregates.
Check yourself: 1. Which view is derived by combining events of one request? 2. Which view is derived by summarising many events over time? 3. Put alert, trace, dashboard and log event in debugging order.
Words you will meet
| Word | Meaning |
|---|---|
| Observability | Understanding a running system from metrics, logs and traces |
| Log | An unchangeable list of time-stamped events |
| Structured event | A log record made of named fields |
| Work unit | A request or a message handled by a service |
| Sanitise | Remove sensitive data before logging |
| Correlation id | A value copied through all events of one request |
| Sampling | Recording only a share of events |
| Trace | The record of one request across services |
| Span | One timed step in a trace |
| Collector | A service that gathers events or spans |