CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 8 of 8 · 2 min read

Exercises

Easy

Easy 1: Nines to minutes (20 minutes)

Compute the downtime allowed per month (30 days) for 99 percent, 99.9 percent and 99.99 percent.

Done when: three numbers in minutes, with the working shown.

Hint

Minutes in a month is 30 times 24 times 60.

Easy 2: Find the SPOFs (20 minutes)

List ten components of a system you know. For each, say what happens if it fails and mark the single points of failure.

Done when: a table of ten rows with a SPOF column.

Hint

Ask "what happens if this is gone for an hour?"

Easy 3: Four silences (20 minutes)

For each of the four reasons a client hears nothing, say what the client can safely assume.

Done when: four rows, each with a safe assumption or "nothing".

Hint

The client cannot tell whether the work was done.

Medium

Medium 1: Leak arithmetic (30 minutes)

A service leaks 2 MB every 500 requests and has 16 GB of memory. Find how long it runs at 1 million and at 50 million requests a day before memory runs out.

Done when: two times in hours, with the working shown.

Hint

Leak per request is 2 MB divided by 500.

Medium 2: Trace a cascade (30 minutes)

Three replicas each have a limit of 120 TPS and a total load of 300 TPS. One fails. Trace the next five minutes, assuming clients retry twice.

Done when: a timeline table with loads and the point where the cascade starts or stops.

Hint

Remember that retries multiply the load.

Medium 3: Risk table (30 minutes)

Build a risk table with at least eight failures of a system you know. Score each and rank them.

Done when: a ranked table and a note on the top two.

Hint

Cheap fixes for high scores come first.

Hard

Hard 1: Pool exhaustion (45 minutes)

A service has a pool of 40 connections. A dependency starts hanging on 10 percent of calls and the service gets 200 calls per second. Estimate how fast the pool empties and what users see.

Done when: a calculation and a timeline marked as made up.

Hint

Calls stuck per second times the time they stay stuck.

Hard 2: Break the cascade (45 minutes)

Using your Medium 2 timeline, list three changes that would have stopped the cascade. Say which later page of this level covers each.

Done when: three changes, each with the new timeline.

Hint

Think of timeouts, limits and removing load.

Hard 3: Pre-mortem (50 minutes)

Pick a system and write a story of its worst outage in the first person, a year from now. List the failure causes from this page that appear in it.

Done when: a one-page story and a table tying each event to a cause.

Hint

Start from the end of the story and work back.