CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 6 of 12 · 2 min read

What the example found

Detection recall and precision by type, the Luhn effect, minimisation by task, and erasure with and without lineage.

How to read the detection table: each row is one PII type in the 20-text labelled corpus; the last row totals every type together.

Type Found / labelled False positives
Email 3 / 4 0
Phone 3 / 4 0
SSN-format 2 / 2 0
Card (Luhn) 2 / 2 0
IPv4 1 / 1 0
Titled name 2 / 3 0
IBAN (no rule) 0 / 1 0
Total 13 / 17 (recall 76%) 0 (precision 100%)

What to notice: the IBAN row is 0 of 1. This is not because the detector made a mistake. It is because no rule for that type exists at all. A missing rule and a wrong rule produce the identical symptom, a miss. That is exactly why recall has to be measured rather than assumed from "the detector didn't flag it. That means there must be nothing there.".

How to read the Luhn-effect note and the minimisation table: the first line is what happens to precision with the Luhn check removed; the table is fields sent per task out of the 12-field record.

Without the Luhn check, an order number, a tracking number and a block of digits inside an IBAN were wrongly redacted as card numbers. This added 3 false positives (precision drops from 100% to 81%).

Task Fields sent (of 12) Personal fields sent
Order status 2 0
Refund 3 0
Reply 3 0

What to notice: every task sends zero personal fields. That is not because the detector caught anything. It is because minimisation never gave the detector a reason to be involved. The fields simply were not part of what was sent.

How to read the erasure table: each row is a different method for erasing one customer's data from a store holding 4 related items for that customer.

Erasure method Copies of the customer left (of 4)
Delete raw record only 3
Follow lineage 0 (other customer kept)

What to notice: "follow lineage" reaching exactly 0 while also correctly leaving the other customer's record untouched is the two-sided proof the method needs to pass. It has to delete everything that belongs to the request and nothing that doesn't.

{{svg:flow|Source record, minimise, tokenise, model and traces, boundary detokenisation.}} {{svg:results|Detection recall by type, erasure with and without lineage, and minimisation.}}