Part 7 of 8 · 2 min read
The whole picture
Each layer catches what the one before it missed.
Closing the level: how the pages fit together
| Layer | Question it answers | Page |
|---|---|---|
| Understand failures | How do small faults grow? | 1 |
| Protect each call | How long do I wait, and when do I try again? | 2 |
| Stop bad calls | When do I stop calling, and how do I keep failures in one place? | 3 |
| Protect the service | How do I refuse or slow work I cannot do? | 4 |
| Break cascades | How do I cut chains and recover safely? | 5 |
| Know the limits | Where is my capacity and does it hold up? | 6 |
| Survive big losses | What if a zone, region or the data is gone? | 7 |
| Notice and repair | Which parts are sick, and who fixes them? | 8 |
No single layer is enough. Timeouts without breakers still wait. Breakers without bulkheads still share pools. Redundancy without health checks sends traffic to dead copies. Health checks without a fail-open rule can take everything out. The skill is to see the chain from trigger to outage and cut it in more than one place.
Interview angle
Answer in your head first, then open it.
How does a load balancer know a server is bad?
It polls a health endpoint, and removes servers that return errors or time out.
What kinds of health check are there?
Liveness, local resource checks and dependency checks. Dependency checks are risky because a shared failure can mark every server unhealthy.
What if every server fails the check?
A smart balancer treats the check as unreliable and ignores it ("fail open"), so traffic still flows.
What is a watchdog?
A monitor that restarts a degraded process. It needs limits and logging so it cannot cause restart loops.
How do you design a resilient service?
Stateless instances, redundancy across zones, timeouts and retries with backoff, breakers, bulkheads, shedding, health checks, and tested recovery.
Check yourself: 1. Which layer in the table handles a data centre fire? 2. Which layer would you add first to a system with none? 3. Why does the skill lie in cutting a chain in more than one place?
Words you will meet
| Word | Meaning |
|---|---|
| Health endpoint | An address a balancer polls to see if a server is fit |
| Liveness | The process is running and answers |
| Local health | Own resources are within limits |
| Dependency health | Needed remote services are reachable |
| Fail open | Ignore a check that looks unreliable and keep sending traffic |
| Watchdog | A monitor that restarts a degraded process |
| Let it crash | Restart instead of repairing a strange state |
| Steady state | The normal range of key measurements |
| Error budget | The amount of failure an SLO allows |
| Self-healing | Detecting and repairing faults without a person |