Part 4 of 8 · 2 min read
Collection
Two ways to move metrics.
The system has five parts: data collection, transmission, storage, alerting and visualisation.
Pull model. Dedicated collectors fetch metrics from each service over HTTP, usually from an endpoint such as /metrics. The collector needs the list of endpoints. A fixed file is hard to keep right as servers come and go, so use service discovery (Level 1), such as etcd or ZooKeeper. Services register, and the collector learns of changes.
One collector cannot cover thousands of servers, so there is a pool. To stop two collectors from pulling the same server, use a consistent hash ring (Level 4). Each server name is hashed onto the ring, and each collector owns a range. A server is then pulled by exactly one collector. A worked example (made up): four collectors and six servers. Each server falls in one collector's range, so no duplicates occur.
Push model. An agent on each server collects local metrics and sends them to a collector. The agent can add up counters locally before sending, which cuts traffic. If the collector refuses, the agent may buffer data on disk, but in auto-scaling groups the server may disappear with the buffer. The collector cluster sits behind a load balancer and scales on load.
Neither wins in every case. Prometheus pulls, and CloudWatch and Graphite receive pushes.
| Pull | Push | |
|---|---|---|
| Debugging | Easy: ask any endpoint at any time | Harder: silence may be a network fault |
| Health check | A failed pull shows a server is down | Missing data may mean many things |
| Short jobs | May finish before being pulled | Fit well |
| Firewalls | Need open paths to each server | Easier: outgoing only |
| Authenticity | Only known servers are pulled | Needs whitelists or login |
| Serverless | Hard | Often the only way |
Large companies often support both.
Read it as
Pull is a teacher walking round the class asking each pupil for their homework. Push is every pupil posting it to the teacher.
Check yourself: 1. How does the collector learn which servers to pull from? 2. How does consistent hashing avoid duplicate pulls? 3. Which model suits a short batch job, and why?