CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 4 of 12 · 2 min read

A reversible token keeps the agent useful without the value ever leaving the vault

Redacting a value destroys information the agent needs; a stable token hides the value while keeping that information intact.

Think of a coat-check ticket. The ticket itself says nothing about which coat is yours. But it reliably gets you back the same coat every time. The counter, not the ticket, holds the real mapping.

pseudonymisation replaces an identifier with a stable token while the real mapping sits in a vault kept outside the prompt, logs and traces, which lets the model still reason about "the same person mentioned twice" without ever seeing the real value. Redaction instead removes the value outright, which is simpler and irreversible but throws away that same-entity information for good.

A worked example. The same email address tokenised twice in the same vault produces the identical token both times (because the token is derived from the value itself, not from when it was seen), while a different vault (a different key) produces a different token for that same address -- so a token never means anything outside the vault that minted it. Detokenisation can be restricted by type: a send-email tool can be allowed to recover <EMAIL_...> tokens specifically, while every other token type in the same text stays opaque to it.

The key lines. From code/mini-privacy/privacy.py:

# from code/mini-privacy/privacy.py
def _token(self, typ, value):
    digest = hmac.new(self.key, f"{typ}|{value}".encode(), hashlib.sha256).hexdigest()
    for n in range(6, len(digest) + 1):              # 6 hex characters normally; longer only if two values collide
        tok = f"<{typ}_{digest[:n]}>"
        if self.map.get(tok, value) == value:
            return tok
    raise RuntimeError("token space exhausted")

Read it as: the token is an HMAC of the type and value, keyed by the vault's own secret. It is truncated to 6 hex characters, unless that would collide with a different value already mapped to the same prefix. The same (type, value) pair always produces the same token within one vault. A different vault's key produces an unrelated string. That is exactly the "stable within a vault, meaningless outside it" property the worked example relies on.

Pseudonymisation, redaction

Pseudonymisation replaces an identifier with a token, with the mapping kept separately. Redaction removes or masks the value, irreversibly.

Example: choosing a token over outright redaction for an email address that a send-email tool will later need is the entire design decision in one sentence -- redaction would make that tool's job impossible, while a token kept in a vault the tool can query solves it without ever putting the real address in a prompt or a log.