Part 2 of 8 · 2 min read
Scope and numbers
Ten million series, constant writes.
Think of the dashboard of a power station. Thousands of gauges report every few seconds. Engineers watch a few graphs, and an alarm sounds when a reading leaves its safe range. The station must keep recording even when a screen is off.
Questions and the answers assumed here (from the book's sample talk).
- For whom? Internal use by one large company.
- Which metrics? Operations data such as processor load, memory, disk space and requests per second. Business figures are left out.
- Scale? The company has 100 million daily users, and runs 1,000 pools of servers with 100 machines in each.
- Retention? 1 year.
- Resolution? Keep every raw point for a week. Then keep one point a minute for a month. After that, keep one point an hour until the year ends.
- Where do alerts go? To email, to phones, to a paging service and to web hooks.
- Logs and tracing? Out of scope.
Requirements.
- R1. Scales with metrics and alerts.
- R2. Low query delay for dashboards and alerts.
- R3. Reliable, so no critical alert is missed.
- R4. Flexible, so new tools can be added.
Numbers. 1,000 pools × 100 machines × 100 metrics per machine is 10 million time series. Suppose each series reports every 10 seconds (made up). That is 10,000,000 / 10 = 1,000,000 data points per second. At 16 bytes a point (8 for the time, 8 for the value, made up), that is 16 MB per second, or 1.38 TB a day. Seven days of raw data is about 9.7 TB.
Roll-ups cut the old data. At 1-minute resolution, 10,000,000 / 60 = about 166,667 points per second, which is 230 GB a day, and 6.9 TB for 30 days. At 1-hour resolution it is about 2,778 points per second, 3.84 GB a day, and 1.4 TB for a year.
The key point: writes are heavy and constant, and reads are bursty, since people open dashboards and alerts check rules at set times.
Time series
A time series is a list of values, each with the time it was measured, for one named thing.
Example: the CPU load of web server 3 at 20:00, 20:00:10 and 20:00:20 is one time series.
Check yourself: 1. How many series does the example have? 2. Why do roll-ups save so much space? 3. Why is the write load called constant but the read load bursty?