CSE Roadmap

Common tier · about 3 hours · 9 parts

Part 3 of 9 · 3 min read

Three ways to find a service

A fixed file, DNS, and a registry, with a worked example of heartbeats and expiry.

Think of three ways to find a restaurant. A printed list in a drawer is cheap but old. A phone book is shared by everyone but updates slowly. A live map app is always current but needs a service behind it.

The book lists the same three choices for finding the servers in a pool.

Method How it works Strength Weakness
Static file A configuration file lists all addresses Simple Painful to keep up to date
DNS One name returns several addresses Standard, shared by all Cached answers can be stale (TTL, page 1)
Data store A registry service holds live entries Most flexible More complexity to run

The registry is the one used in large systems.

Service registry

A service registry is a database of the running instances of each service, kept up to date by the instances themselves.

Example: the registry holds payment → 10.0.1.4:8080, 10.0.1.9:8080, 10.0.2.3:8080.

The usual steps:

  1. Register. When an instance starts, it tells the registry its name and address.
  2. Heartbeat. It sends a small message every few seconds to say it is alive.
  3. Expire. If heartbeats stop, the registry removes the entry after a limit.
  4. Lookup. A caller asks for the instances of a service and picks one.
  5. Deregister. On a clean shutdown, the instance removes itself.

A worked example. Heartbeats are sent every 5 seconds, and an entry expires after 3 missed beats (15 seconds).

  • A server crashes at time 0, just after a beat.
  • The last beat was at time 0. Misses are counted at 5, 10 and 15 seconds.
  • The entry is removed at 15 seconds.
  • During those 15 seconds, callers may still be sent to a dead address. If the service has 5 instances, about 1 call in 5 goes to the dead one, unless callers retry on another instance.

The trade-off sits in the numbers. Beats every 1 second with expiry after 3 misses give 3 seconds of exposure, but five times more heartbeat traffic. With 1,000 instances, 1 beat a second is 1,000 messages per second, not a problem. With 100,000 instances it is a heavy load, and shorter intervals need care.

A second question is who does the lookup.

  • Server-side discovery. The caller talks to a load balancer, and the balancer consults the registry. The caller knows one stable name. Page 3 describes this design.
  • Client-side discovery. The caller queries the registry itself and picks an instance. This avoids one extra hop and one shared balancer, but every client needs the logic. The book describes a variant, the sidecar, where a helper process on the same machine does the lookup and balancing for the caller. Level 7 covers it.

Common mistake

Letting a dead entry stay for minutes because the expiry was left at a default. Check the interval and limit, and make callers retry another instance when a call fails.

Check yourself:

  • With a beat every 4 seconds and expiry after 3 missed beats, how long can a dead address remain listed?
  • What is the difference between server-side and client-side discovery?