CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 5 of 8 · 2 min read

Pipeline

Protect the database.

If the database is down, collectors have nowhere to write and data is lost. The book adds a message queue such as Kafka (page 1 of this level) between collectors and storage. Collectors write to the queue, and consumers or a stream processor such as Flink write to the database. Benefits: the parts are decoupled, and data waits safely in the queue during a database outage. Partitioning by metric name lets one consumer aggregate one metric, and important metrics can have their own partitions.

An interviewer may say a queue is heavy to run. Some systems, such as Facebook's in-memory Gorilla database, stay available for writes without a queue, by being built for partial failure.

Where to aggregate.

Place How Good Bad
Agent Sum counters per minute Less traffic Simple logic only
Ingestion pipeline Stream processor Much less written Late events are hard, raw data lost
Query time On request No data lost Slower queries

Query service. A thin layer of servers between the database and the dashboards and alerts, perhaps with a cache. It lets the database or dashboard tool change later. With a good time-series database and tools that connect directly, this layer may not be needed.

Saving space.

  • Encoding. Store the first timestamp, then only the differences. Timestamps 1000, 1010, 1020, 1029, 1046 become the base 1000 and deltas 10, 10, 9, 17. Deltas are small numbers and compress well.
  • Downsampling. Convert fine data to coarser data as it ages, following the retention rules. Ten-second points are averaged into 30-second points. A worked example (made up): readings 4, 6 and 8 average to 6.
  • Cold storage. Very old data that is rarely read moves to cheap storage.

Downsampling

Downsampling replaces many fine-grained points with fewer summary points, such as an average per minute.

Example: six 10-second readings become one 1-minute reading.

Check yourself: 1. What does the queue protect against? 2. Which aggregation place loses raw data? 3. Why do deltas compress better than timestamps?