CSE Roadmap

Core tier · about 3 hours · 9 parts

Part 7 of 9 · 1 min read

Making qualities measurable

How to turn a quality into a number, and how to keep it true as the system changes.

A quality that cannot be measured cannot be managed. "The system should be fast" is a hope. "99 out of 100 searches finish in under 400 milliseconds" is a target.

Here are measures for common qualities:

Quality A measure
Performance Response time at p50 and p99; requests per second at peak
Scalability Requests per second handled as users double
Elasticity Time to add servers after a sudden jump; extra capacity at peak
Availability Percentage of time up, such as 99.9 percent
Recoverability Time to restore service after a serious failure
Maintainability Time for a new engineer to ship a first change
Security Number of known open problems, time to fix them

Measuring once is not enough. Systems change, and qualities slowly decay. A common method is an automatic test that checks a quality on every change. Such a test is sometimes called a fitness function.

Fitness function

A fitness function is an automatic check that tests whether a quality of the design still holds. It runs regularly, so that a slow loss of the quality is caught early.

Example: a test that fails the build if the page's response time at p99 rises above 400 milliseconds.

A worked example. A team sets a target: the home page must load in under 400 ms at p99. A test runs on every release. In March, a new feature raises p99 to 450 ms. The test fails, the release stops, and the team fixes the slowdown before any user sees it. Without the test, the page would have become slower with each release, and no one would know which change caused the problem.

Check yourself:

  • Turn "the app must be reliable" into a measurable statement.
  • Why does a one-time measurement fail to protect a quality?