Sensitive data is cleaned at ingestion, with Presidio, before anything is indexed

Context

Some of the documents our AI application answers from carry personal and sensitive data. Retrieval does not know what is sensitive: whatever sits in the index can be retrieved into a prompt and repeated in an answer. Embeddings do not hide it either. Morris and colleagues showed in 2023 that short texts can be recovered almost word for word from their embeddings. So the place to remove sensitive values is before chunking and embedding, in the ingestion pipelines.

Why Presidio

It runs where the documents already are. Presidio is an open-source library, so it runs inside our ingestion pipelines as one more step before chunking. Documents are cleaned where they are processed instead of being sent to another service for detection.

Detection that can be extended. Presidio's analyzer combines a named-entity model with pattern recognisers, checksums and the context words around a match. Custom recognisers cover identifiers specific to one organisation, next to the built-in ones for names, emails, phone numbers and the like.

Masking chosen per entity. Its anonymizer replaces, redacts, masks, hashes or encrypts each finding. Replacing a name with a typed placeholder such as <PERSON> keeps the sentence readable for retrieval, where redaction leaves a hole the embedding model has to guess around.

What it costs

Detection is probabilistic, so it has to be tuned rather than trusted: measure what gets masked and what gets missed on representative documents, and adjust recognisers and thresholds. The two kinds of error are not equal. A false positive masks a product or a place name and quietly makes retrieval worse; a miss leaves the value in place. So favour recall for the entities that matter most, and watch precision through retrieval quality.

Every document also pays for one more processing step, and masking at ingestion is final for the index: a use case that needs an original value reads it from the source system, under that system's own permissions.

When it is the wrong layer

If some users must see the unmasked text, masking at ingestion cannot serve them. Their access has to be enforced at retrieval instead, with per-document permissions on an index that holds the original.