Part 7 of 8 · 3 min read
Testing failure
Prove the defences work before an outage proves they do not.
Testing failure on purpose
UDS is frank about testing: nobody can list every way a large distributed application may fail, so tests give only best-effort guarantees. They cover failures developers can imagine, not the emergent behaviour that shows up in production.
Two practices from UDS help.
- Canary releases. Release a new version to a small number of production instances first. The goal is to surface problems as soon as possible, before they can do widespread damage. If the health checks pass, roll out to the rest. With several regions, start with a low-traffic one.
- Chaos testing. Inject controlled failures into production to "shake the tree". UDS says it shows whether the dependencies can hold the promised service level and whether the safety mechanisms really work. BM (Level 7) describes Netflix's Chaos Monkey, which switches off random machines, and Google's yearly disaster recovery exercises that simulate events such as earthquakes. A game day is a planned session in which teams simulate a failure and respond to it.
UDS also says end-to-end tests should not hurt other users of a shared environment, which needs good fault isolation, such as rate limits.
Chaos testing
Chaos testing means deliberately causing failures in a running system, under control, to confirm that it copes.
Example: switch off one of six instances at midday and check that p99 stays within its target.
A safe way to start (common practice): begin in staging, then in production at a quiet hour, with a small blast radius, a clear stop rule and the right people watching. Expand only when each step is boring.
Interview angle
Answer in your head first, then open it.
How do you find a service's capacity?
A load test with open-loop traffic, rising in steps, watching p99, errors and resources. The capacity is just below where targets are missed.
Why percentiles and not the average?
The tail is what the slowest and often most valuable users see, and tails grow with the number of calls per request.
How many instances do you need?
Peak load divided by safe capacity per instance, plus room for losing a failure unit and for a rolling release.
How do you know your defences work?
Failure drills and chaos tests in a controlled way, and canary releases.
Check yourself: 1. What does a canary release limit? 2. Why can tests not prove a system is resilient? 3. What should every chaos experiment have before it starts?
Words you will meet
| Word | Meaning |
|---|---|
| Capacity | The highest load that still meets targets |
| Goodput | Requests answered correctly and in time per second |
| Percentile | The value that a given share of requests beat |
| Tail latency | The response times at high percentiles |
| Head-of-line blocking | Fast requests held up behind a slow one |
| Tail latency amplification | The chance of a slow call growing with the number of calls |
| SLO / SLA | Target / contract for performance and availability |
| Load test | Generated traffic used to find capacity |
| Canary release | A first release to a few production instances |
| Chaos testing | Injecting controlled failures on purpose |
| Game day | A planned drill that simulates a failure |