Part 6 of 6 · 2 min read
Exercises
Easy
Easy 1: Name the state (20 minutes)
For five moments in a breaker's life (first call, after the tenth failure, 10 seconds into open, after the wait, after a good test call) say which state it is in.
Done when: five answers.
Hint
The breaker only tests from the open state after the wait.
Easy 2: Pool arithmetic (20 minutes)
A shared pool has 60 connections for 3 downstream services. Show what happens if one service hangs, then redo it with 3 pools of 20.
Done when: two short walk-throughs.
Hint
Count free connections for the healthy services.
Easy 3: Choose the fallback (20 minutes)
For a shop, give a graceful fallback for each of: recommendations down, reviews down, payment down, search down.
Done when: four fallbacks, with one marked "must fail".
Hint
Ask which feature is the reason the user came.
Medium
Medium 1: Draw the timeline (30 minutes)
Using a threshold of 3 failures in 5 seconds and an open time of 20 seconds, draw the states over 90 seconds when the dependency fails from second 10 to second 50.
Done when: a timeline with each transition marked.
Hint
Do not forget the half-open tests that fail.
Medium 2: Code a breaker (30 minutes)
Write a small class with the three states, a threshold, a window and an open time. Test it with a fake service that fails for a while.
Done when: a working class and a test output.
Hint
Keep a list of failure times and drop old ones.
Medium 3: Count the virtual partitions (30 minutes)
A pool has 12 instances. Count the pairs and the triples. Compare the chance two random clients share an identical pair and an identical triple.
Done when: counts and two probabilities.
Hint
Pairs from 12 are 12 times 11 divided by 2.
Hard
Hard 1: Rescue the story (45 minutes)
Rewrite BM's ad service story with your three fixes in place. Give the thread count over the first five minutes before and after.
Done when: two tables marked as made up.
Hint
The threads tied up equal calls per second times the wait.
Hard 2: Breaker settings (45 minutes)
A dependency has a normal error rate of 0.5 percent and 200 calls per second. Choose a threshold and window that rarely trips by chance but trips within 5 seconds of a real outage. Explain the trade-off.
Done when: settings with a calculation and a note on false trips.
Hint
At 200 calls per second, 0.5 percent is 1 failure a second.
Hard 3: Design for a bad tenant (50 minutes)
A multi-tenant API has 40 instances and 500 tenants. One tenant sends a request that crashes the server it lands on. Design the bulkhead and the retry rules so that it harms as few other tenants as possible.
Done when: a design with numbers for fixed and virtual partitions.
Hint
A crashing request retried on another instance moves the harm.