CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 2 of 8 · 2 min read

Scope and numbers

Ten million series, constant writes.

Think of the dashboard of a power station. Thousands of gauges report every few seconds. Engineers watch a few graphs, and an alarm sounds when a reading leaves its safe range. The station must keep recording even when a screen is off.

Questions and the answers assumed here (from the book's sample talk).

  1. For whom? Internal use by one large company.
  2. Which metrics? Operations data such as processor load, memory, disk space and requests per second. Business figures are left out.
  3. Scale? The company has 100 million daily users, and runs 1,000 pools of servers with 100 machines in each.
  4. Retention? 1 year.
  5. Resolution? Keep every raw point for a week. Then keep one point a minute for a month. After that, keep one point an hour until the year ends.
  6. Where do alerts go? To email, to phones, to a paging service and to web hooks.
  7. Logs and tracing? Out of scope.

Requirements.

  • R1. Scales with metrics and alerts.
  • R2. Low query delay for dashboards and alerts.
  • R3. Reliable, so no critical alert is missed.
  • R4. Flexible, so new tools can be added.

Numbers. 1,000 pools × 100 machines × 100 metrics per machine is 10 million time series. Suppose each series reports every 10 seconds (made up). That is 10,000,000 / 10 = 1,000,000 data points per second. At 16 bytes a point (8 for the time, 8 for the value, made up), that is 16 MB per second, or 1.38 TB a day. Seven days of raw data is about 9.7 TB.

Roll-ups cut the old data. At 1-minute resolution, 10,000,000 / 60 = about 166,667 points per second, which is 230 GB a day, and 6.9 TB for 30 days. At 1-hour resolution it is about 2,778 points per second, 3.84 GB a day, and 1.4 TB for a year.

The key point: writes are heavy and constant, and reads are bursty, since people open dashboards and alerts check rules at set times.

Time series

A time series is a list of values, each with the time it was measured, for one named thing.

Example: the CPU load of web server 3 at 20:00, 20:00:10 and 20:00:20 is one time series.

Check yourself: 1. How many series does the example have? 2. Why do roll-ups save so much space? 3. Why is the write load called constant but the read load bursty?