CSE Roadmap

Core tier · about 3 hours · 6 parts

Part 5 of 6 · 4 min read

Aborts and retries

Why a failed transaction is simply tried again, and the ways that can go wrong.

Aborts and retries

Think of a form at a government office. If the clerk finds a mistake, the form is returned and you hand it in again. That works when handing it in twice does no harm and the clerk was only briefly confused. It fails if the first copy was in fact accepted.

The book says a key feature of a transaction is that it can be aborted and safely retried if an error occurred. ACID databases follow this philosophy: if the database is in danger of breaking atomicity, isolation or durability, it prefers to abandon the transaction rather than leave it half finished. Not all systems do this. Leaderless stores (Level 4, module 3) work on a "best effort" basis: they do what they can and, on error, do not undo what is done, so recovering is up to the application.

The book also notes that popular object-relational mappers such as Rails ActiveRecord and Django do not retry aborted transactions. The error becomes an exception, the user's input is thrown away and an error message is shown. It calls this a shame, since aborts exist to enable safe retries.

Retrying is simple and effective but not perfect. The book lists five problems:

  1. The commit succeeded but the acknowledgement was lost. The client thinks it failed and retries, so the work happens twice unless the application has its own deduplication. This is why idempotency keys (Level 1, module 2) matter.
  2. Overload. If the error comes from overload, retrying makes it worse. Limit the number of retries, use exponential backoff and treat overload errors differently.
  3. Permanent errors. Retry only transient errors: deadlock, isolation violation, brief network faults, failover. After a constraint violation, a retry is pointless.
  4. Side effects outside the database. An email sent during a transaction is not unsent when it aborts, and a retry may send it again. To make several systems commit or abort together, two-phase commit can help (module 9).
  5. The client fails during the retry. Whatever data it was trying to write is lost.

A worked example of backoff. A transaction aborts on a deadlock. Retry after 100 ms, then 200 ms, then 400 ms, up to 5 retries, with a random extra 0 to 50 percent so that many clients do not retry at the same instant. The longest total wait is 100 + 200 + 400 + 800 + 1,600 = 3,100 ms plus the random part. If all 5 retries fail, the client reports an error.

A worked example of the duplicate. A payment transaction commits in 20 ms. The reply is lost. The client times out at 1 second and retries. Without protection, the customer is charged twice. With an idempotency key (a unique request id stored with the payment), the second attempt finds the key already used, does not repeat the payment and returns the original result.

Common mistake

Retrying every failure at once and without limit. Under overload it multiplies the traffic. Retry only transient errors, with a limit and growing delays.

Interview angle

Answer in your head first, then open it.

What does ACID mean?

Atomicity (all or nothing), consistency (the application's invariants), isolation (concurrent transactions do not interfere), durability (committed data survives crashes). Say that C belongs to the application and that durability is risk reduction.

Do you need transactions in a scalable system?

Often yes, for a few important flows. The belief that they cannot scale is too simple. Say which operations need multi-record atomicity (orders, payments).

How do you make a retry safe?

Make the transaction abortable, add an idempotency key for the commit-but-lost-reply case, retry only transient errors with backoff and a limit.

What is the difference between a transaction and a compare-and-set?

One object against several, and what each prevents.

Check yourself:

  • Why is retrying after a lost acknowledgement dangerous without extra protection?
  • Which errors are worth retrying and which are not?

Words you will meet

Word Plain meaning
Transaction A group of reads and writes treated as one unit
Commit The transaction succeeds and its writes become permanent
Abort (roll back) The transaction fails and its writes are undone
ACID Atomicity, consistency, isolation, durability
BASE "Basically available, soft state, eventual consistency": vague, roughly "not ACID"
Atomicity All of a transaction's writes happen or none
Invariant A statement about the data that must always be true
Isolation Concurrent transactions do not interfere
Serializability The result equals running the transactions one after another
Durability Committed data is not forgotten
Compare-and-set Write only if the value is still what the client expects
Multi-object transaction A transaction that changes several records at once
Idempotency key A unique id that lets the server detect a repeated request