CSE Roadmap

Core tier · about 6 hours · 13 parts

Part 5 of 13 · 2 min read

Turn every fixed failure into a trace the fix is checked against forever

A bug you fixed once but didn't test for can come back the moment someone touches the code again.

Think of a vaccine record. Once you have had a specific illness and recovered, the record exists precisely so nobody has to guess later whether you are still vulnerable to that exact strain.

In plain words: a regression case is a stored failing input, or trace, that must pass after the fix and in every later version of the code — converting a one-time debugging session into a permanent, replayable check means the specific bug, once found, can never silently come back without a test failing immediately.

A worked example. After fixing the retry-storm rule's precision problem, the specific failing trace that exposed the plan-loop false positive becomes a stored test case: a trace is replayed with stored tool outputs (so it doesn't call any live service), and the test asserts the fixed detector no longer flags it as a retry storm while still correctly flagging genuine retry storms. If someone later edits retry_storm and accidentally removes the errors_only check, this stored case fails immediately rather than the bug quietly reappearing in production.

The key lines. From code/mini-debug/detect.py:

# from code/mini-debug/detect.py
def to_regression_case(t):
    """A failing trace turned into a replayable test case (keep failing traces as a regression corpus)."""
    return {"task": t.task, "expect_no": t.fault, "steps": [(s.kind, s.name, s.status, tuple(sorted(s.args))) for s in t.spans]}

Read it as: the stored case keeps expect_no: t.fault as an explicit assertion, not just the raw trace. A regression case is not merely "replay this input and see what happens." It is "replay this input and specifically check that this particular fault is gone." That is what makes it catch a reintroduced bug, instead of just producing output someone has to eyeball.

Regression case

A regression case is a stored failing input (or trace) that must pass after the fix and in all later versions.

Example: storing the retry-storm trace with expect_no: "retry_storm" turns one debugging session into a permanent tripwire — exactly the kind of artifact an interviewer wants to hear you produced at the end of your bug story, not just a fix with no evidence it will stay fixed.