fix(ml): detect PII past the model window and stop inflating confidence

[mazzasaverio]

Stacked on #57, which it contains. Review or merge that one first; the diff against it is src/pseudonymize/backends/ml/onnx.py and its tests only.

Two defects in LocalONNXPIIBackend. Both make the backend report something other than what it actually found, and the first one silently leaves personal data in the output.

Personal data past the model window was silently dropped

The shipped tokenizer.json configures truncation at 512 tokens:

>>> tokenizer.truncation
{'max_length': 512, 'stride': 0, 'strategy': 'longest_first', 'direction': 'right'}

so tokenizer.encode quietly discarded every token past it. Measured on the cached DistilBERT with "Contact Maria Rossi in Milan." at the end of a growing block:

Preceding text	Detections on main
none	Maria Rossi PERSON, Milan LOCATION
~100 words	Maria Rossi, Milan
~300 words	Rossi only
~600 words	none

No exception, no warning, no statistic. The caller receives a document that looks sanitized. Roughly two kilobytes is one page of a PDF, or one long email.

Truncation is now turned off and the block is split into overlapping token windows sized from the tokenizer's own limit, falling back to the config's max_position_embeddings and then to 512. Detections are shifted back to absolute offsets, and a span found in two overlapping windows collapses to its higher-confidence copy. The same entity is now found at 36 KB:

Block size	After
484 B	Maria Rossi 0.998, Milan 0.959
3.6 KB	Maria Rossi 0.997, Milan 0.971
11 KB	Maria Rossi 0.994, Milan 0.952
36 KB	Maria Rossi 0.995, Milan 0.913

The reported confidence was manufactured

dynamic_boost_threshold = policy.minimum_confidence * 0.01
...
calibrated_conf = policy.minimum_confidence + (raw_conf - dynamic_boost_threshold) / (
    1.0 - dynamic_boost_threshold
) * (1.0 - policy.minimum_confidence)

A runner-up label beat O at one hundredth of the policy floor, and every surviving span was then rescaled from that point onto the floor. Two consequences:

• minimum_confidence could not filter anything. Clearing the boost guaranteed clearing the floor, by construction.
• It was inverted. A stricter floor produced a lower boost threshold, so asking for more confidence returned more detections.

The same weak prediction, on main:

min_conf 0.5  -> ('Apollo', PERSON, 0.5055)
min_conf 0.99 -> ('Apollo', PERSON, 0.9901)

The model's actual probability for that span is 0.016.

Confidence is now the model's own probability, so raising minimum_confidence can only ever remove detections. Recall is tuned by a separate explicit entity_threshold (default 0.5): the absolute probability a runner-up label must reach to beat O. window_overlap_tokens is exposed alongside it, and both are validated.

For your call

Honest probabilities cost recall at the default 0.8 floor, which is calibrated for deterministic detectors emitting 0.9–1.0 rather than for a probabilistic model. One existing test needed its policy lowered to 0.5 for the second of two John Smith mentions (true probability 0.717) to be detected.

The ai4privacy F1 will fall. That is the previously reported number being corrected rather than a regression: 0.10.0's "calibration" gain came from this rescaling. Whether the ML backend deserves a different default floor from the rules backend is a real question, and worth deciding on its own rather than inside this fix.

Verification

Six new tests: windowing at three block sizes, window coverage and overlap, confidence monotonicity under two floors, threshold-controlled recall, and argument validation.

256 passed, 1 skipped
coverage 95.46%   (main measures 95.31% locally, below its own 95.36% gate)
ruff, mypy        clean

🤖 Generated with Claude Code

https://claude.ai/code/session_012JiX3zWeEC28kmy5KmAXvf