CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 7 of 8 · 3 min read

Testing failure

Prove the defences work before an outage proves they do not.

Testing failure on purpose

UDS is frank about testing: nobody can list every way a large distributed application may fail, so tests give only best-effort guarantees. They cover failures developers can imagine, not the emergent behaviour that shows up in production.

Two practices from UDS help.

  • Canary releases. Release a new version to a small number of production instances first. The goal is to surface problems as soon as possible, before they can do widespread damage. If the health checks pass, roll out to the rest. With several regions, start with a low-traffic one.
  • Chaos testing. Inject controlled failures into production to "shake the tree". UDS says it shows whether the dependencies can hold the promised service level and whether the safety mechanisms really work. BM (Level 7) describes Netflix's Chaos Monkey, which switches off random machines, and Google's yearly disaster recovery exercises that simulate events such as earthquakes. A game day is a planned session in which teams simulate a failure and respond to it.

UDS also says end-to-end tests should not hurt other users of a shared environment, which needs good fault isolation, such as rate limits.

Chaos testing

Chaos testing means deliberately causing failures in a running system, under control, to confirm that it copes.

Example: switch off one of six instances at midday and check that p99 stays within its target.

A safe way to start (common practice): begin in staging, then in production at a quiet hour, with a small blast radius, a clear stop rule and the right people watching. Expand only when each step is boring.

Interview angle

Answer in your head first, then open it.

How do you find a service's capacity?

A load test with open-loop traffic, rising in steps, watching p99, errors and resources. The capacity is just below where targets are missed.

Why percentiles and not the average?

The tail is what the slowest and often most valuable users see, and tails grow with the number of calls per request.

How many instances do you need?

Peak load divided by safe capacity per instance, plus room for losing a failure unit and for a rolling release.

How do you know your defences work?

Failure drills and chaos tests in a controlled way, and canary releases.

Check yourself: 1. What does a canary release limit? 2. Why can tests not prove a system is resilient? 3. What should every chaos experiment have before it starts?

Words you will meet

Word Meaning
Capacity The highest load that still meets targets
Goodput Requests answered correctly and in time per second
Percentile The value that a given share of requests beat
Tail latency The response times at high percentiles
Head-of-line blocking Fast requests held up behind a slow one
Tail latency amplification The chance of a slow call growing with the number of calls
SLO / SLA Target / contract for performance and availability
Load test Generated traffic used to find capacity
Canary release A first release to a few production instances
Chaos testing Injecting controlled failures on purpose
Game day A planned drill that simulates a failure