CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 4 of 12 · 2 min read

A fix that learns the attacks you showed it overfits to them

A defence built from the exact wording of known attacks is a defence against that wording, not against the underlying goal.

Think of a spam filter trained only on emails that say "WIN A FREE PRIZE." It blocks that exact phrase beautifully. It does nothing at all against "You've been selected for a complimentary reward," even though both are the same scam.

Held-out attacks are attacks not used to design the fix, used only to test it. The gap between a fix's performance on the attacks it was built from versus attacks it has never seen is exactly what reveals whether a fix is structural (blocking what an action does, regardless of wording) or just a pattern match on wording it happened to be shown.

A worked example. A blocklist of word 3-grams learned from the three training phrasing families cuts their success rate to 5.2% -- but on the three held-out families it never saw, the rate is 9.3%, against an undefended 22.3% baseline on those same held-out families. A gate that blocks the action itself (not the wording) instead gives 9.0% on training families and 9.3% on held-out ones -- almost identical, because the gate never looked at phrasing to begin with, so there was never a gap for held-out attacks to expose.

The key lines. From code/mini-redteam/redteam.py:

# from code/mini-redteam/redteam.py
def blocked(self, atk):
    if self.blocklist and any(g in re.sub(r"\s+", " ", atk["text"].lower()) for g in self.blocklist):
        return True
    a = atk["action"]
    if self.gate >= 1 and a in ("email", "delete_order", "refund_large"):
        return True
    if self.gate >= 2 and a in ("read_other", "refund_small"):
        return True
    return False

Read it as: the blocklist check reads atk["text"], the attack's wording, while the gate check reads atk["action"], what the attack is trying to do. Those are two different fields on the same object. That is the entire structural difference between a fix that overfits to phrasing and one that does not. One depends on a field an attacker can freely rewrite. The other does not.

Held-out attacks

Held-out attacks are attacks not used to design the fix, used only to test it.

Example: the blocklist's jump from 5.2 percent to 9.3 percent between training and held-out families is held-out evaluation doing exactly its job -- revealing a weakness in the fix that the training numbers alone would have hidden completely.