Part 3 of 9 · 3 min read
Three ways to find a service
A fixed file, DNS, and a registry, with a worked example of heartbeats and expiry.
Think of three ways to find a restaurant. A printed list in a drawer is cheap but old. A phone book is shared by everyone but updates slowly. A live map app is always current but needs a service behind it.
The book lists the same three choices for finding the servers in a pool.
| Method | How it works | Strength | Weakness |
|---|---|---|---|
| Static file | A configuration file lists all addresses | Simple | Painful to keep up to date |
| DNS | One name returns several addresses | Standard, shared by all | Cached answers can be stale (TTL, page 1) |
| Data store | A registry service holds live entries | Most flexible | More complexity to run |
The registry is the one used in large systems.
Service registry
A service registry is a database of the running instances of each service, kept up to date by the instances themselves.
Example: the registry holds payment → 10.0.1.4:8080, 10.0.1.9:8080, 10.0.2.3:8080.
The usual steps:
- Register. When an instance starts, it tells the registry its name and address.
- Heartbeat. It sends a small message every few seconds to say it is alive.
- Expire. If heartbeats stop, the registry removes the entry after a limit.
- Lookup. A caller asks for the instances of a service and picks one.
- Deregister. On a clean shutdown, the instance removes itself.
A worked example. Heartbeats are sent every 5 seconds, and an entry expires after 3 missed beats (15 seconds).
- A server crashes at time 0, just after a beat.
- The last beat was at time 0. Misses are counted at 5, 10 and 15 seconds.
- The entry is removed at 15 seconds.
- During those 15 seconds, callers may still be sent to a dead address. If the service has 5 instances, about 1 call in 5 goes to the dead one, unless callers retry on another instance.
The trade-off sits in the numbers. Beats every 1 second with expiry after 3 misses give 3 seconds of exposure, but five times more heartbeat traffic. With 1,000 instances, 1 beat a second is 1,000 messages per second, not a problem. With 100,000 instances it is a heavy load, and shorter intervals need care.
A second question is who does the lookup.
- Server-side discovery. The caller talks to a load balancer, and the balancer consults the registry. The caller knows one stable name. Page 3 describes this design.
- Client-side discovery. The caller queries the registry itself and picks an instance. This avoids one extra hop and one shared balancer, but every client needs the logic. The book describes a variant, the sidecar, where a helper process on the same machine does the lookup and balancing for the caller. Level 7 covers it.
Common mistake
Letting a dead entry stay for minutes because the expiry was left at a default. Check the interval and limit, and make callers retry another instance when a call fails.
Check yourself:
- With a beat every 4 seconds and expiry after 3 missed beats, how long can a dead address remain listed?
- What is the difference between server-side and client-side discovery?