CSE Roadmap

Core tier · about 5 hours · 8 parts

Part 5 of 8 · 2 min read

Exactly once

Retry, but charge once.

The worst failure is charging a customer twice. The aim is that a payment happens exactly once. The book splits this into two simpler promises: at least once (retry until it works) and at most once (never do it twice, using idempotency).

Retry strategies.

  • Immediate retry: resend at once. Aggressive, and can overload a service.
  • Fixed interval: wait the same time between tries.
  • Incremental: wait a little longer each time.
  • Exponential backoff: double the wait after each failure.
  • Cancel: stop, for permanent errors.

A worked example (made up). With waits of 1, 2, 4, 8 and 16 seconds, five retries take 1 + 2 + 4 + 8 + 16 = 31 seconds in total. Use exponential backoff when the problem is not likely to fix itself soon. Add a random wiggle so many clients do not retry together, and give users a clear error.

Idempotency. An operation is idempotent if repeating it gives the same result as doing it once. The client creates an idempotency key, often a unique id, and sends it in a header. The server remembers keys. A repeated key gets the stored result, not a second charge. If two requests with the same key arrive at the same moment, one runs and the other gets a "too many requests" reply.

How to build it. Use the database's unique key. Insert a row with the key as primary key. If the insert works, the request is new. If it fails because the key exists, it is a repeat. In an online shop, the id of the shopping cart at the time of the click is a natural key.

Second scenario. The PSP succeeded, but the reply never reached the system, so the user presses pay again. Nothing about the order has changed, so the nonce and token that go to the PSP are the ones used before. The PSP uses the token as its own idempotency key and recognises the repeat, returning the first result.

Failed requests. Failures that may succeed later (a timeout) go to a retry queue. After too many tries, a message goes to a dead letter queue, where people inspect it. Failures that cannot succeed (invalid input) are stored with the error.

State. Keep a definite, saved state for each payment at every stage, in an append-only table, so after any crash the system knows whether to retry, refund or wait.

Idempotency key

An idempotency key is a unique value sent with a request, so that repeating the request produces the first result and no second effect.

Example: clicking "pay" twice with key K-77 charges once and returns the same receipt both times.

Check yourself: 1. How are at-least-once and at-most-once combined? 2. How does a unique key in the database give idempotency? 3. What happens to a message that fails every retry?