Comparison of tools that remove personal data
How reliably open tools remove personal data from documents before they go to an AI tool. Tested on 1,000 synthetic UK documents that none of the tools had seen.
| Tool | Documents with a leak | Identifiers | Names, addresses, organisations | Precision |
|---|---|---|---|---|
| pii-roundtrip | 0.2% | 100.0% | 99.98% | 99.8% |
| GLiNER Guard Omni | 58.3% | 95.4% | 88.9% | 94.8% |
| Presidio | 61.1% | 92.5% | 58.5% | 79.0% |
| OpenAI privacy-filter | 81.1% | 95.7% | 84.4% | 92.9% |
| Prompt Anonymizer | 100.0% | 68.1% | 51.5% | 99.0% |
| Regular expressions with checksums | 100.0% | 90.7% | 0.0% | 100.0% |
Held-out test set 7, pii-roundtrip version 0.5. Each tool was run once, with the settings from its documentation, on the same 1,000 documents. Some tools were not built for whole documents or for UK identifiers; the figures describe this test, not the tools in general. Counting a partly hidden value as a leak too, pii-roundtrip leaves one in 0.2% of documents.
How to read the table
- Documents with a leak: the share of documents in which at least one real value remained after redaction. Lower is better. This is the main measure: one surname left behind can be enough to identify a person.
- Identifiers: the share of NHS and National Insurance numbers, IBANs and card numbers that were hidden.
- Names, addresses, organisations: the share of these values that were hidden.
- Precision: the share of masked fragments that really contain personal data. It shows whether a tool simply masks everything.
Why documents and not short messages
Public datasets for this task hold short messages. A real document mentions the same person several times: in the header, in the body and in the signature, sometimes with initials or in another form. A tool that misses one of these mentions leaks the person. The test documents are contracts, references, letters, HR files and discharge summaries.
How the test was protected from tuning
The documents were generated after the last engine changes and published with their hashes, the scoring rules and the run plan before the first run. Each tool ran once, and every run is logged. A tool is never scored on a dataset built by its own authors.
Limits
The benchmark was run by the author of pii-roundtrip. The documents are synthetic and come from this project's own generator, which pii-roundtrip was developed on and the other tools never saw. Code, data, results and run logs are public so that anyone can repeat the test: full report.