CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 5 of 8 · 1 min read

Controlling log cost

Say less, but keep the signal.

UDS offers several ways to keep the bill in check.

  • Levels. Use levels such as debug, info, warning and error, with a dynamic setting that decides which are emitted. Operators turn up detail when investigating and turn it down when not.
  • Sampling. Log only one in every n events. Rank by expected signal: failures should be sampled more often than successes.
  • Rate limits at the collector. Adding nodes multiplies volume, and someone may commit a bug that logs far too much. Collectors need to rate-limit, as hosted services usually do with a quota.
  • Aggregate instead. Compute metrics in memory from the events and send only those. This loses the chance to drill down later.

A made-up calculation. A service handles 5,000 requests per second, and each event is 1 KB. Logging all of them gives 5,000 × 1 KB = 5 MB per second, about 432 GB per day (5 MB × 86,400 s). Sampling 1 percent of successes (say 4,900 of the 5,000) and all 100 failures gives 49 + 100 = 149 events per second, about 12.9 GB per day.

Sampling

Sampling means recording only a chosen share of events, so that cost falls while the overall pattern stays visible.

Example: keep every failed request and one in a hundred successful ones.

Check yourself: 1. Why sample failures more often than successes? 2. What does a collector's rate limit protect against? 3. What is the price of replacing logs with in-memory metrics?