Part 7 of 9 · 1 min read
Making qualities measurable
How to turn a quality into a number, and how to keep it true as the system changes.
A quality that cannot be measured cannot be managed. "The system should be fast" is a hope. "99 out of 100 searches finish in under 400 milliseconds" is a target.
Here are measures for common qualities:
| Quality | A measure |
|---|---|
| Performance | Response time at p50 and p99; requests per second at peak |
| Scalability | Requests per second handled as users double |
| Elasticity | Time to add servers after a sudden jump; extra capacity at peak |
| Availability | Percentage of time up, such as 99.9 percent |
| Recoverability | Time to restore service after a serious failure |
| Maintainability | Time for a new engineer to ship a first change |
| Security | Number of known open problems, time to fix them |
Measuring once is not enough. Systems change, and qualities slowly decay. A common method is an automatic test that checks a quality on every change. Such a test is sometimes called a fitness function.
Fitness function
A fitness function is an automatic check that tests whether a quality of the design still holds. It runs regularly, so that a slow loss of the quality is caught early.
Example: a test that fails the build if the page's response time at p99 rises above 400 milliseconds.
A worked example. A team sets a target: the home page must load in under 400 ms at p99. A test runs on every release. In March, a new feature raises p99 to 450 ms. The test fails, the release stops, and the team fixes the slowdown before any user sees it. Without the test, the page would have become slower with each release, and no one would know which change caused the problem.
Check yourself:
- Turn "the app must be reliable" into a measurable statement.
- Why does a one-time measurement fail to protect a quality?