Part 2 of 8 · 1 min read
What monitoring is for
Find failures that hurt users, and show system health.
UDS says monitoring has two jobs. The first is to detect failures that affect users in production and send alerts to the people who must fix them. The second is to give a high-level view of health through dashboards.
Early monitoring was black-box: it said only whether a service was up or down. Developers then began to add measurements inside their code, which is white-box monitoring. UDS credits Etsy's tool statsd with making application-level measurement normal.
Both are still used. Black-box checks run scripts from outside, in the regions where users are, send test requests to the real endpoints and record the time and success. They catch what is invisible from inside, such as connectivity problems, and APIs that users rarely call. White-box monitoring finds the cause of known hard failures before users are hurt. UDS's rule of thumb: if you cannot design a hard failure mode away, add monitoring for it.
Black-box and white-box monitoring
Black-box monitoring tests a service from the outside, as a user would. White-box monitoring uses measurements the service reports about its own inside.
Example: a script that calls the login page every minute from three cities (black-box). A counter of failed password checks inside the service (white-box).
Check yourself: 1. What are the two main jobs of monitoring? 2. Why do black-box checks run from the regions where users are? 3. What does UDS say to do with a failure you cannot design away?