Part 5 of 8 · 2 min read
Pipeline
Protect the database.
If the database is down, collectors have nowhere to write and data is lost. The book adds a message queue such as Kafka (page 1 of this level) between collectors and storage. Collectors write to the queue, and consumers or a stream processor such as Flink write to the database. Benefits: the parts are decoupled, and data waits safely in the queue during a database outage. Partitioning by metric name lets one consumer aggregate one metric, and important metrics can have their own partitions.
An interviewer may say a queue is heavy to run. Some systems, such as Facebook's in-memory Gorilla database, stay available for writes without a queue, by being built for partial failure.
Where to aggregate.
| Place | How | Good | Bad |
|---|---|---|---|
| Agent | Sum counters per minute | Less traffic | Simple logic only |
| Ingestion pipeline | Stream processor | Much less written | Late events are hard, raw data lost |
| Query time | On request | No data lost | Slower queries |
Query service. A thin layer of servers between the database and the dashboards and alerts, perhaps with a cache. It lets the database or dashboard tool change later. With a good time-series database and tools that connect directly, this layer may not be needed.
Saving space.
- Encoding. Store the first timestamp, then only the differences. Timestamps 1000, 1010, 1020, 1029, 1046 become the base 1000 and deltas 10, 10, 9, 17. Deltas are small numbers and compress well.
- Downsampling. Convert fine data to coarser data as it ages, following the retention rules. Ten-second points are averaged into 30-second points. A worked example (made up): readings 4, 6 and 8 average to 6.
- Cold storage. Very old data that is rarely read moves to cheap storage.
Downsampling
Downsampling replaces many fine-grained points with fewer summary points, such as an average per minute.
Example: six 10-second readings become one 1-minute reading.
Check yourself: 1. What does the queue protect against? 2. Which aggregation place loses raw data? 3. Why do deltas compress better than timestamps?