CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 7 of 8 · 2 min read

Trade-offs

What to say.

Trade-offs and what to say

Choice Gain Cost
Time-series database Built for the load and queries A specialised tool to run
Pull Easy debugging, known sources Hard for serverless, needs open paths
Push Fits short jobs and firewalls Needs authentication
Queue in front Survives database outage More to operate
Aggregate in the pipeline Less stored Raw data lost
Roll-ups Cheap long retention Lost detail on old data
Alert manager Fewer, better alerts More rules to maintain

Reliability of the monitor itself. A monitoring system must outlive what it watches. If it runs in the same data centre and that fails, there is no alert about the failure. Run monitors in another zone, and have a simple "heartbeat" check that pages if the monitor itself goes quiet.

Interview angle

Answer in your head first, then open it.

What kind of database?

A time-series database.

Pull or push?

Know both, and say it depends. Large companies support both.

How do you avoid data loss?

A queue between collectors and storage.

How do you keep storage small?

Delta encoding, downsampling, cold storage.

How do alerts avoid noise?

A waiting time in the rule, de-duplication, grouping and inhibition.

One follow-up to try. "Add log monitoring and tracing." A good reply: these are different data. Logs are text records and are searched, not summed, so they go into a search store such as an ELK-style stack. Traces follow one request across services (Level 9). Reuse the same ideas of collectors, a queue and storage, but with a different database for each, and join them by a shared request id.

Check yourself: 1. Who watches the monitor? 2. Why are logs stored differently from metrics? 3. What links a log line to a trace?

Words you will meet

Word Meaning
Time series Timestamped values for one named thing
Label A key and value that tags a series
Cardinality The number of distinct values of a label
Collector A server that gathers metrics
Downsampling Replacing fine data with coarser summaries
Alert rule A condition that raises an alert
Alert manager A service that merges and routes alerts
Inhibition Silencing minor alerts while a major one is active