CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 3 of 8 · 2 min read

Metrics

Numbers over time, labelled so they can be sliced.

A metric is a number measured over a time interval and stored as a time series: a list of samples, each a number with a timestamp, such as the count of requests handled. Modern systems let you attach labels, which are key and value pairs such as region, cluster or service. Each distinct set of labels is effectively a different metric. That makes it easy to slice the data without writing separate code for each cut.

A service should emit metrics about its load, its internal state, and the availability and speed of the services it calls. UDS shows how many questions one small handler raises: was the cache hit, how long was the item cached, did the remote call fail or time out, how long did it take, how big is the cache. Answering them takes deliberate work by developers.

Events or pre-aggregated metrics? One way to count failures is to send an event for every failure. That costs a lot, since load on the backend grows with the number of events, and a query such as "failures in North Europe over a month" must scan possibly trillions of events. The cheaper way is to pre-aggregate: sum or average the samples over fixed periods (for example, one hour) and store only those. UDS gives a small series: 561 failures in the hour from 00:00, 42 from 01:00, 61 from 02:00. The agent in each process can aggregate too, which cuts bandwidth, compute and storage. The cost is lost flexibility. A metric stored by the hour cannot later be seen in 5-minute pieces, since the original events are gone.

Metric

A metric is a measurement stored as numbers over time, usually with labels that say where it came from.

Example: requests_failed{region="eu-north", service="checkout"} with one value per minute.

Check yourself: 1. What does a label add to a metric? 2. Why is storing an event for every failure costly? 3. What is lost when metrics are pre-aggregated?