CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 2 of 12 · 2 min read

A regex catches what it has a rule for, and nothing else

Treating a detector's silence as proof of absence is the single most dangerous habit in redaction.

Think of a metal detector at an airport gate. It reliably catches the metal shapes it was tuned for. It walks straight past a ceramic blade. That is not because the ceramic blade is safe. It is because the detector was never built to see it.

personal data (PII) is information that can identify a person, alone or combined with other data, and a detector built from regular expressions finds only the shapes it has a rule for. Precision (of what it flagged, how much was truly personal data) can be high while recall (of what was truly personal, how much it flagged) stays well short of complete. That is because every undetected shape is a silent miss, not a visible error.

A worked example. Against a 20-text labelled set written for the example, the regex rules found 13 of 17 labelled items with zero false positives: recall 76%, precision 100%. The four misses were an obfuscated email ("alice at example dot test"), a name with no title ("Alice Moreau" with no "Dr." or "Mr." in front), a spelled-out phone number, and an IBAN, a type the detector has no rule for at all. Without a Luhn check on card numbers, three more false positives appeared -- an order number, a tracking number, and a block of digits inside the IBAN -- dropping precision to 81%, because those digit strings happen to be the right length for a card number.

The key lines. From code/mini-privacy/privacy.py:

# from code/mini-privacy/privacy.py
def luhn_ok(digits):
    s = [int(c) for c in digits][::-1]
    return (sum(s[0::2]) + sum((2 * d - 9 if 2 * d > 9 else 2 * d) for d in s[1::2])) % 10 == 0

Read it as: the Luhn check is one extra pass over a digit string that already matched the length pattern for a card number. It is the difference between "looks like it could be a card number" and "passes the checksum real card numbers satisfy." That is exactly what separates a 13-digit order number from an actual card.

Personal data, PII, precision and recall of a detector

Personal data is information that can identify a person, alone or combined with other data; PII is the common US-style term. Precision is, of what a detector flagged, how much was truly personal data; recall is, of what was truly personal, how much it flagged.

Example: "76 percent recall, 100 percent precision" is not a passing grade for the detector -- it's a measured statement that roughly a quarter of the real personal data in this test slipped through untouched, which is the number that should drive every other design decision in this module.