Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-reliability.
Done when: you report success and primary load for "retry x3" and "breaker + fallback".
Easy 2: Compound failure (30 minutes)
Compute the success of a 12-call agent run when each call succeeds 99 percent of the time, then 99.9 percent.
Done when: you explain what that implies for retries and durability.
Hint
0.99 to the power 12 is about 0.886.
Easy 3: Choose retryable errors (30 minutes)
Classify ten errors from your agent (timeouts, 429, 500, 400, tool validation) as retry or not.
Done when: each has a reason.
Medium
Medium 1: Breaker tuning (3 hours)
Sweep window, threshold and cooldown and plot success and primary load.
Done when: you show the trade-off between false opens and slow detection.
Hint
open too easily and you shed healthy traffic; too slowly and you waste calls.
Medium 2: Slow outage (4 hours)
Change the outage to 8-second responses instead of fast errors.
Done when: timeouts, concurrency limits and bulkheads change the results and you explain how.
Hint
a slow failure ties up workers for the whole timeout, which is why timeouts and pools matter.
Medium 3: Gateway (4 hours)
Write a small gateway that routes through primary and fallback providers with the layers above.
Done when: a log line records which path served each request.
Hard
Hard 1: Fault injection (8 hours)
Test your real agent with injected outages, delays and 429s. (I did not run this, so your numbers are the ground truth.)
Done when: you report success, p95 latency and calls to the failing service with each layer toggled.
Hard 2: Fallback quality (6 hours)
Run your eval suite on the fallback model. (I did not run this.)
Done when: you decide which tasks may use the fallback and which should escalate to a human.
Hint
record which tasks would pass at the fallback's quality.
Hard 3: Correlated failure (6 hours)
Make the fallback fail when the primary fails (shared region). (I did not run this.)
Done when: you design a third option and show the new success rate.