Part 6 of 12 · 2 min read
What the example found
Detection recall and precision by type, the Luhn effect, minimisation by task, and erasure with and without lineage.
How to read the detection table: each row is one PII type in the 20-text labelled corpus; the last row totals every type together.
| Type | Found / labelled | False positives |
|---|---|---|
| 3 / 4 | 0 | |
| Phone | 3 / 4 | 0 |
| SSN-format | 2 / 2 | 0 |
| Card (Luhn) | 2 / 2 | 0 |
| IPv4 | 1 / 1 | 0 |
| Titled name | 2 / 3 | 0 |
| IBAN (no rule) | 0 / 1 | 0 |
| Total | 13 / 17 (recall 76%) | 0 (precision 100%) |
What to notice: the IBAN row is 0 of 1. This is not because the detector made a mistake. It is because no rule for that type exists at all. A missing rule and a wrong rule produce the identical symptom, a miss. That is exactly why recall has to be measured rather than assumed from "the detector didn't flag it. That means there must be nothing there.".
How to read the Luhn-effect note and the minimisation table: the first line is what happens to precision with the Luhn check removed; the table is fields sent per task out of the 12-field record.
Without the Luhn check, an order number, a tracking number and a block of digits inside an IBAN were wrongly redacted as card numbers. This added 3 false positives (precision drops from 100% to 81%).
| Task | Fields sent (of 12) | Personal fields sent |
|---|---|---|
| Order status | 2 | 0 |
| Refund | 3 | 0 |
| Reply | 3 | 0 |
What to notice: every task sends zero personal fields. That is not because the detector caught anything. It is because minimisation never gave the detector a reason to be involved. The fields simply were not part of what was sent.
How to read the erasure table: each row is a different method for erasing one customer's data from a store holding 4 related items for that customer.
| Erasure method | Copies of the customer left (of 4) |
|---|---|
| Delete raw record only | 3 |
| Follow lineage | 0 (other customer kept) |
What to notice: "follow lineage" reaching exactly 0 while also correctly leaving the other customer's record untouched is the two-sided proof the method needs to pass. It has to delete everything that belongs to the request and nothing that doesn't.
{{svg:flow|Source record, minimise, tokenise, model and traces, boundary detokenisation.}} {{svg:results|Detection recall by type, erasure with and without lineage, and minimisation.}}