Part 2 of 8 · 3 min read
Timeouts
Never wait forever.
Think of a phone call where the other side never speaks. After a while you hang up. How long you wait is up to you, but "forever" is not a plan.
UDS makes the point sharply: a call with no timeout tells the code that it is one hundred percent sure the call will succeed. Timeouts limit and isolate failures, so they stop failures from cascading. They apply to network calls, but also to waiting for a connection from a pool and to waiting on locks.
Timeout
A timeout is the longest time a caller waits for a reply before giving up on the call.
Example: a caller waits at most 2 seconds for a price lookup. If nothing arrives, it reports an error.
The book shows that many tools have no timeout unless asked. The browser's XMLHttpRequest has a default of zero, which means none. The fetch API had no way to set a timeout when it first came out. Python's requests library and Go's HTTP client default to waiting forever. Modern Java and .NET clients come with defaults, and .NET's 100 seconds is lax but better than none. The rules of thumb:
- Always set a timeout on a network call.
- Be wary of libraries that make network calls or hold pools but expose no timeout setting.
- If you write a library, set a sensible default and let users change it.
Timeouts matter on the client side as much as on the server. A browser can open only so many sockets to one host. Requests that never return use them all up.
How to pick the value. UDS suggests basing it on the false timeout rate you can accept. A false timeout is a call that would have succeeded if it had been given more time. To have about 0.1 percent false timeouts, set the timeout at the 99.9th percentile of the call's response time, which you measure. (Level 0 explained percentiles.)
A worked example (made up). A price service answers 99 percent of calls in under 120 ms, 99.9 percent in under 400 ms and the slowest 0.1 percent in up to 6 seconds. A 400 ms timeout cuts off about 1 call in 1,000 and never ties up a thread for more than 400 ms. No timeout lets the slowest calls hold threads for 6 seconds or more. At 500 calls per second, 0.1 percent is 0.5 calls per second stuck for 6 seconds, so about 3 threads are held on average, which a pool can take. If the dependency starts to hang on every call, the same arithmetic gives 500 threads a second tied up, and the pool is empty within seconds.
You also need to measure what happens at each call: how long it took, the status code, and whether a timeout fired. Without this, production problems are hard to debug. UDS advises wrapping each remote call in a library that sets the timeout and records these facts, so no one has to remember. A reverse proxy placed next to the process (a sidecar, as on Level 7 page 5) can do the same job from outside.
Check yourself: 1. Why does a call with no timeout hide a bet? 2. Which percentile gives about 0.1 percent false timeouts? 3. Name two things other than a network call that need a timeout.