Part 2 of 9 · 2 min read
The front door for a pool of servers
Why one address in front of many servers raises capacity and availability.
Think of a bank with five counters. Customers join one queue, and a greeter sends each one to a free counter. If one counter closes, customers still get served. Nobody needs to know which counter did the work.
A load balancer plays the greeter for requests.
Load balancer
A load balancer (often written LB) receives requests at one address and passes each one to a server chosen from a group. The group is called the pool.
Example: users call shop.example.com. The load balancer hands each request to one of four web servers.
The book "Understanding Distributed Systems" explains how this works. The LB owns one or more virtual IP addresses (VIPs), and each VIP is tied to a pool. Clients see the VIP only. They never learn the addresses of the servers behind it. In the book "System Design Interview", the servers behind the LB use private addresses that cannot be reached from the internet, so users can only come in through the LB.
This gives three benefits.
- More capacity. If one server handles 1,000 requests per second, then four of them handle about 4,000. The book describes this as the quickest and cheapest way to scale a service that holds no state of its own.
- Higher availability. The LB notices a failed server and stops using it. Users keep getting answers.
- Easy change. Servers can be added or removed without clients noticing, since clients only know the VIP.
A worked example. A shop needs to serve 2,500 requests per second at the peak. One server handles 1,000.
- Servers needed for the load: 2,500 ÷ 1,000 = 2.5, so 3.
- One more is added so that a single failure is survivable: 4 servers. This habit is called N + 1.
- With all four up, capacity is 4,000. With one lost, it is 3,000, still above 2,500.
Stateless service
A stateless service keeps nothing about a user between two requests. Any server in the pool can answer any request.
Example: a server that reads the price of product 42 from a database and returns it. It does not remember the visitor.
The book warns about one catch. More servers also mean more pressure on the things behind them. If every server uses the same database, the database becomes the limit, and adding servers only strains it more. Level 2 and Level 3 deal with that.
Read it as
A load balancer turns "which server?" from a question the client asks into a decision the system makes.
Check yourself:
- Why do clients only need to know one address?
- Why is a fourth server added when three are enough for the load?