CSE Roadmap

Core tier · about 9 hours · 13 parts

Part 5 of 13 · 4 min read

Unit tests, and the limits of testing one caller at a time

A single cashier watching one queue never notices six people reaching for the same last item on a shelf at the same moment. A test that calls a function once, from one thread, has the same blind spot.

In plain words: a test with one caller cannot prove a concurrent write is safe; only a test with several callers racing for the same row can.

The service's create accepts an idempotency key so a retried request returns the same ticket instead of filing it twice. The naive way to implement that is to look the key up, and insert if it is missing. That has a gap between the look and the write. The deliberately broken version of the server (racy_idem) takes that path:

# from code/mini-build/service.py
key = idempotency_key if "no_idem" not in self.bugs else None
if key is not None:
    if "racy_idem" in self.bugs:
        existing = self.conn.execute("SELECT * FROM tickets WHERE idem = ?", (key,)).fetchone()
        if existing:
            return {"ticket": row(existing), "created": False}
        self.hook()
        cur = self.conn.execute("INSERT INTO tickets (title, body, priority, customer, created, updated, idem) VALUES (?,?,?,?,?,?,?)", (title, body, priority, customer, now, now, key))
        self.conn.commit()
        return {"ticket": row(self._get_row(cur.lastrowid)), "created": True}
    cur = self.conn.execute("INSERT INTO tickets (title, body, priority, customer, created, updated, idem) VALUES (?,?,?,?,?,?,?) ON CONFLICT DO NOTHING",
                            (title, body, priority, customer, now, now, key))

Read it as: the racy_idem branch does the look and the write as two separate steps. self.hook() sits in the gap so a test can pause there on purpose. The correct path below it does both in one statement. It lets the database's own unique index decide who wins.

A test that calls create twice with the same key, one after another, passes on both versions — it never lands in the gap. The race check instead starts six threads, each with its own database connection to a shared file, and uses the pause point to make the race happen every time, not by luck:

Version Rows created for one key (6 callers) Distinct answers
single atomic write with a unique index 1 1
"look first, then insert" with no unique index 6 6

The only real fix is to let the database enforce uniqueness (INSERT ... ON CONFLICT DO NOTHING on a unique index); a unit test around a single caller can never see this, which is why the check uses threads.

Race condition, idempotency key

A race condition is a bug that only appears when two or more operations run at overlapping times and their order decides the outcome. An idempotency key is a value the caller invents once per logical request, so repeating the same key is recognised as a retry rather than a new action. Example: six threads inserting with the same idempotency key produce six rows under "look first, then insert", and one row under a unique index.

Two more checks in this layer are worth reading closely. Input as data, not as code: the search check passes zzz') OR 1=1 OR (title LIKE ' as the query and expects 0 results. The server's search puts the text in a bound parameter. A version that pastes the text into the SQL string returned every ticket. Visibility: a customer's search returns exactly her four tickets. Asking for someone else's ticket returns the same text as asking for a number that does not exist. Here is one complete state-machine check, as written:

@check("unit")
def close_and_reopen_follow_the_state_machine(bugs):
    d = app.Env(bugs).desk
    t = d.close("alice", 1, "Reset the password by hand.")["ticket"]
    assert t["status"] == "closed" and t["resolution"] == "Reset the password by hand."
    expect_error(lambda: d.close("alice", 1, "again"), "BAD_STATE", "`ticket_reopen`")
    assert d.reopen("alice", 1)["ticket"]["resolution"] is None
    expect_error(lambda: d.reopen("alice", 1), "BAD_STATE", "already open")
    expect_error(lambda: d.close("alice", 3, ""), "INVALID", "resolution")