CSE Roadmap

Common tier · about 4 hours · 8 parts

Part 4 of 8 · 2 min read

Collection

Two ways to move metrics.

The system has five parts: data collection, transmission, storage, alerting and visualisation.

Pull model. Dedicated collectors fetch metrics from each service over HTTP, usually from an endpoint such as /metrics. The collector needs the list of endpoints. A fixed file is hard to keep right as servers come and go, so use service discovery (Level 1), such as etcd or ZooKeeper. Services register, and the collector learns of changes.

One collector cannot cover thousands of servers, so there is a pool. To stop two collectors from pulling the same server, use a consistent hash ring (Level 4). Each server name is hashed onto the ring, and each collector owns a range. A server is then pulled by exactly one collector. A worked example (made up): four collectors and six servers. Each server falls in one collector's range, so no duplicates occur.

Push model. An agent on each server collects local metrics and sends them to a collector. The agent can add up counters locally before sending, which cuts traffic. If the collector refuses, the agent may buffer data on disk, but in auto-scaling groups the server may disappear with the buffer. The collector cluster sits behind a load balancer and scales on load.

Neither wins in every case. Prometheus pulls, and CloudWatch and Graphite receive pushes.

Pull Push
Debugging Easy: ask any endpoint at any time Harder: silence may be a network fault
Health check A failed pull shows a server is down Missing data may mean many things
Short jobs May finish before being pulled Fit well
Firewalls Need open paths to each server Easier: outgoing only
Authenticity Only known servers are pulled Needs whitelists or login
Serverless Hard Often the only way

Large companies often support both.

Read it as

Pull is a teacher walking round the class asking each pupil for their homework. Push is every pupil posting it to the teacher.

Check yourself: 1. How does the collector learn which servers to pull from? 2. How does consistent hashing avoid duplicate pulls? 3. Which model suits a short batch job, and why?