Part 3 of 8 · 1 min read
Logs
An append-only record of what happened.
A log is an unchangeable list of time-stamped events. An event can be free text, but UDS prefers structured events: key and value pairs in a format such as JSON. A made-up event:
{ "timestamp": 1614438079, "serviceRegion": "EastUs2", "failureCount": 1 }
Logs come from your services and from outside software such as message brokers, proxies and databases. They are usually written to disk files, rotated now and then, and forwarded asynchronously by an agent to a collector, such as an ELK stack or AWS CloudWatch Logs.
Logs help trace a symptom, such as a crash, back to its cause. They also explain the long tail that averages and percentiles hide, and they can say why one particular request failed. A metric cannot.
UDS lists the drawbacks, and the only advantage over metrics is that logs are easy to emit: - A badly used logging library adds delay, especially when writing to disk or the console blocks the request. - If excessive logging fills the disk, the instance can degrade. At best logging stops. At worst the instance stops working if it needs the disk. - Storing and processing a mass of events is costly, even in binary form, because of the many dimensions. - The signal-to-noise ratio is poor, since events are fine-grained and specific to one service.
Structured log event
A structured log event is one record made of named fields, so it can be searched and filtered by machine.
Example: {"request_id":"r-42","user":"u-7","status":500,"ms":120} can be filtered for status 500, whereas the sentence "request failed" cannot.
Check yourself: 1. Why is a log better than a metric for explaining one failed request? 2. How can heavy logging damage a service? 3. Why are structured events preferred?