Part 8 of 9 · 3 min read
The balancer is a part too
The balancer can fail and become a bottleneck, so it needs a plan of its own.
The balancer is a part too
Think of a single door into a stadium. It protects the building, but if it jams, nobody gets in.
The book states the drawback plainly. All traffic goes through the load balancer. If it fails, the service behind it cannot be reached. It is also one more thing to run and scale. So the balancer needs the same care as any other part.
Common answers:
- A pair, one active. Two balancers share one virtual address. If the active one stops answering, the standby takes the address within seconds. This is called active-passive failover.
- Several active. Several balancers each take a share, for example by DNS or by network routing. If one fails, the others carry its share, as long as the survivors have spare capacity.
- Managed service. A cloud balancer is run and scaled by the provider, which moves the problem to them.
A worked example of sizing. Peak traffic is 40,000 requests per second. One layer 7 balancer handles 25,000 (made up).
- Two balancers with a perfect split would each carry 20,000.
- If one fails, the other must carry 40,000, which is above 25,000. Failure causes an overload.
- With three balancers, each carries 13,333 at peak. If one fails, the other two carry 20,000 each, which is under 25,000. The design survives one failure.
The rule: plan so that after one failure, the survivors can still carry the whole peak.
Other limits to remember:
- The balancer adds a hop and a little delay to every request.
- It can hide the real client address from the servers. Balancers pass it along in a header, and servers should read it from there.
- Long-lived connections (chat, streaming) hold balancer resources for a long time, so they change the sizing.
Interview angle
Answer in your head first, then open it.
Where does the load balancer go in your drawing, and what happens if it dies?
Place it in front of the stateless servers and say how it is made redundant: a standby with a shared address, or several active ones sized to survive one loss.
Layer 4 or layer 7?
Layer 4 for raw speed and simple spreading. Layer 7 when routing by path or cookie, ending TLS or retrying is needed. Large designs use both.
How does a request get to the nearest region?
Geo DNS returns the address of the nearest healthy region, and regional balancers handle failover inside it.
A server is down. How fast do users stop seeing errors?
Detection time from health checks, plus retries. Give the numbers: interval times failures needed.
Check yourself:
- If a design uses two balancers and each is at 70 percent load, what happens when one fails?
- Name two ways to remove the balancer as a single point of failure.
Words you will meet
| Word | Plain meaning |
|---|---|
| Load balancer (LB) | Spreads requests over a pool of servers |
| Pool | The group of servers behind a balancer |
| VIP | The virtual address that clients see |
| N + 1 | One more server than the load needs |
| Stateless | Remembers nothing between requests |
| Layer 4, layer 7 | The TCP level, and the HTTP level |
| Hash function | Turns an input into a number, always the same one |
| Reverse proxy | A middleman in front of servers |
| TLS termination | Ending the encrypted channel at the proxy |
| Consistent hashing | A way to place keys so that few move when servers change |
| Herding | Many requests chasing one server that looks idle |
| Hotspot | A part that gets far more work than the rest |
| Health check | A test that tells if a server may receive requests |
| Geo load balancing | DNS that returns the nearest region |
| Sidecar | A helper process on the same machine that handles network jobs |