CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 7 of 8 · 2 min read

Where to put the code

One place, not every call.

Where to put the code

UDS advises wrapping each remote call in one library that sets the timeout, applies backoff with jitter, and records metrics. There are libraries in most languages that offer these patterns. The other route is a proxy next to the process that intercepts all outgoing calls. Either way, individual developers do not need to remember to add the protection each time.

The timeout, the retry limit and the backoff cap are linked. A caller's overall timeout must be longer than the sum of its tries and delays, or the outer caller gives up before the inner retries finish. With timeouts of 400 ms, 3 tries and delays of about 1 s and 2 s, the total can reach about 4.2 s. The outer call should therefore allow at least that, or retries should be cut.

Interview angle

Answer in your head first, then open it.

Why set timeouts?

A call without one can hang forever, holding a thread or connection, and can start a cascade.

How do you choose a timeout?

Measure the response time and pick the percentile that matches the false timeout rate you accept.

What is exponential backoff with jitter?

Delays that double up to a cap, with a random part so clients do not retry together.

When should you not retry?

On errors that will not change, and on non-idempotent calls without an idempotency key.

What is retry amplification?

Retries at several levels multiplying the load. Retry at one level only.

Check yourself: 1. Why must the outer timeout exceed the inner retries? 2. Name one benefit of a shared library for remote calls. 3. In the 4.2 second example, how is the total built?

Words you will meet

Word Meaning
Timeout The longest wait for a reply
False timeout A call cut off that would have succeeded
Backoff A growing delay between retries
Jitter A random part in the delay
Retry storm Many clients retrying at the same moment
Retry queue A queue holding failed requests to try again later
Idempotent Safe to repeat with the same effect
Retry amplification Multiplied calls from retries at several levels
Fail fast Give up at once and report the error