Part 5 of 8 · 4 min read
RPC, REST and messages
Three ways to build the calls.
Remote procedure call (RPC) makes a call on another machine look like a local one. Tools with a separate interface definition (SOAP, Thrift, protocol buffers) can generate code for client and server in different languages. Getting started is very fast. BM names the traps.
- Technology coupling. Some RPC ties both sides to one platform. Java RMI binds the client and the server to the JVM.
- Local calls are not remote calls. Hiding the network hides the cost. A local call is nearly free, while a remote one has to pack the data, cross the network and unpack it. BM recalls the first fallacy of distributed computing: the network is reliable. It is not. Failures can be fast, slow or garbled, and the caller must tell a bad request from a failed server.
- Brittleness. Generated stubs mean a new method needs new stubs, and removing an unused field can break callers that never used it. Server and client must then be released together, which BM calls lock-step releases. Binary types become expand-only: they collect fields that nobody dares to remove.
BM does not call RPC terrible. Protocol buffers and Thrift avoid some of the old problems, and many operations suit it. His advice: do not hide the network, and make sure the interface can change without forcing callers to upgrade together.
REST (representational state transfer) is an architectural style inspired by the Web. Its key idea is the resource: something the service knows about, such as a Customer. The server offers a representation of it, such as JSON, and how it is shown is decoupled from how it is stored. REST is most often used over HTTP, which supplies verbs such as GET and POST. A further idea is hypermedia: a response includes links to what the caller can do next, so the caller does not hard-code addresses.
BM lists downsides. A client stub cannot be generated as easily as with RPC. Teams sometimes slip back into sending RPC-style calls through HTTP, or building shared client libraries. HTTP also has overhead, so it is a poor fit when latency must be extremely low. He still calls REST over HTTP a sensible default for service-to-service calls.
One more warning from BM: some web frameworks make it easy to take a database object and expose it directly. This couples the outside to the inside storage. One team he saw delayed real storage until the interface settled, and kept entities in a local file for a while, so that what callers wanted shaped the design and not the table layout.
Messages are the third way. A message broker such as RabbitMQ accepts events from producers and tells subscribers, and can even keep track of what each consumer has seen. It is another system to run, but BM is a fan. Two cautions. First, keep the middleware dumb and the smarts in the services, since vendors tend to pile logic into the broker (the Enterprise Service Bus is his example). Second, async has its own failure modes.
BM tells a story from 2006, a bank's pricing system. Workers took pricing jobs off a queue. One kind of job crashed the worker, the lock timed out, the job went back on the queue, and the next worker crashed on it too. The workers kept dying. The fixes: repair the bug, set a maximum number of retries, and add a "message hospital" (a dead letter queue, Level 6 page 2) with a screen to view and replay bad messages. He also advises correlation ids, which tag a request so it can be traced across services.
Common mistake
Hiding the network so well that nobody remembers each call can fail or be slow. A good interface looks remote to its users, because it is.
Remote procedure call
A remote procedure call (RPC) is a technique where a call to a function on another machine is written like a local call.
Example: customer.find("1234") in the code actually travels over the network to the customer service.
Check yourself: 1. Why does the removal of an unused field break RPC callers but not a tolerant caller? 2. What is a resource in REST? 3. What three defences did the pricing team add after the dying workers?