Part 4 of 8 · 2 min read
Networks and slow processes
A call that never returns is worse than one that fails.
When a client sends a request and hears nothing, it has only two options: keep waiting, or give up. UDS lists four reasons for the silence:
- the server is slow;
- the request was dropped on the way;
- the server crashed while working on it;
- the reply was dropped on the way.
The client cannot tell which one. UDS calls slow network calls the silent killers of distributed systems, because a client that waits too long, or forever, causes problems that are very hard to trace.
Slow processes matter for the same reason. To an outside observer, a very slow process looks much like one that is not running. The common cause is a resource leak. A memory leak slowly fills memory, then the system swaps pages to disk and the garbage collector works harder, which slows everything more. Threads and connections leak too. If a thread makes a blocking call with no timeout and the call never returns, that thread is gone from its pool. The pool has a fixed size. Lose enough threads and none are left. Making the calls asynchronous does not fix it. Modern HTTP clients keep a pool of connections, and a request without a timeout never gives its connection back.
Libraries add risk. Code you did not write uses the same memory, threads and sockets, and you often cannot see how.
Resource leak
A resource leak happens when a program takes a resource, such as memory, a thread or a connection, and never gives it back.
Example: a request that waits forever holds one connection from a pool of 50. After 50 such requests, no connection is left.
Common mistake
Thinking that a slow service is better than a dead one. A dead service fails fast and callers move on. A slow one holds their threads and connections, and the slowness spreads to them.
Check yourself: 1. Name the four reasons a client may hear no reply. 2. Why does a missing timeout lead to a pool running dry? 3. Why does async calling not remove the problem?