Part 7 of 8 · 2 min read
Trade-offs
What to say.
Trade-offs and what to say
| Choice | Gain | Cost |
|---|---|---|
| Time-series database | Built for the load and queries | A specialised tool to run |
| Pull | Easy debugging, known sources | Hard for serverless, needs open paths |
| Push | Fits short jobs and firewalls | Needs authentication |
| Queue in front | Survives database outage | More to operate |
| Aggregate in the pipeline | Less stored | Raw data lost |
| Roll-ups | Cheap long retention | Lost detail on old data |
| Alert manager | Fewer, better alerts | More rules to maintain |
Reliability of the monitor itself. A monitoring system must outlive what it watches. If it runs in the same data centre and that fails, there is no alert about the failure. Run monitors in another zone, and have a simple "heartbeat" check that pages if the monitor itself goes quiet.
Interview angle
Answer in your head first, then open it.
What kind of database?
A time-series database.
Pull or push?
Know both, and say it depends. Large companies support both.
How do you avoid data loss?
A queue between collectors and storage.
How do you keep storage small?
Delta encoding, downsampling, cold storage.
How do alerts avoid noise?
A waiting time in the rule, de-duplication, grouping and inhibition.
One follow-up to try. "Add log monitoring and tracing." A good reply: these are different data. Logs are text records and are searched, not summed, so they go into a search store such as an ELK-style stack. Traces follow one request across services (Level 9). Reuse the same ideas of collectors, a queue and storage, but with a different database for each, and join them by a shared request id.
Check yourself: 1. Who watches the monitor? 2. Why are logs stored differently from metrics? 3. What links a log line to a trace?
Words you will meet
| Word | Meaning |
|---|---|
| Time series | Timestamped values for one named thing |
| Label | A key and value that tags a series |
| Cardinality | The number of distinct values of a label |
| Collector | A server that gathers metrics |
| Downsampling | Replacing fine data with coarser summaries |
| Alert rule | A condition that raises an alert |
| Alert manager | A service that merges and routes alerts |
| Inhibition | Silencing minor alerts while a major one is active |