fix(benchmarks): score precision and recall consistently, and realign the docs

[mazzasaverio]

Stacked on #57, which it contains. The diff against it is benchmarks/, pyproject.toml and the docs.

The release gate from 0.8.0 onwards is a number produced by benchmarks/evaluate_quality.py. That number was not precision or recall.

The scoring was inconsistent on both sides of the ratio

for d_start, d_end in detected_spans:
    matched = False
    for i, (g_start, g_end, _g_label) in enumerate(ground_truth_spans):
        if max(d_start, g_start) < min(d_end, g_end):
            matched = True
            matched_gt.add(i)          # <- marks every overlapping annotation
    if matched:
        true_positives += 1            # <- but credits only one

Three problems:

1. Mismatched units. True positives are counted per detection; false negatives per annotation. One detection covering two adjacent annotations — "John Smith" against separate GIVENNAME and SURNAME masks, which this corpus produces constantly — scores one true positive while removing two from the recall denominator. Recall is inflated by construction.
2. Labels ignored. _g_label is discarded. A detection agreeing on position but not on entity type is a hit, so finding an email where the corpus annotated a surname scores.
3. Correct work punished. Annotations outside SUPPORTED_LABELS are dropped from the truth set, so a correct detection of one becomes a false positive. That rewards suppressing valid detections.

Detections and annotations are now paired one to one, largest overlap first, and a pair must agree on entity type via an explicit corpus-label-to-EntityType map. --span-only restores the label-blind scoring for comparison. Out-of-scope hits are reported on their own line instead of being charged as errors.

What the released code actually scores

Same 0.17.0 code, unchanged, 300 validation rows, ONNX backend enabled:

Scoring	Precision	Recall	F1
As reported in the README	0.9475	0.8910	0.9184
Corrected, --span-only	0.9316	0.7529	0.8328
Corrected, entity types compared	0.8094	0.6536	0.7232

The gap is measurement, not behaviour. The 90% threshold that 0.13.0 declared cleared, and that 0.14.0 onwards were gated against, was never met under scoring that counts one unit on both sides and checks that a detection agrees with what it is credited for.

I have not touched the gate itself in this PR — where the bar belongs now that the numbers mean something is your call, and it interacts with #58 and #59, which both move detection behaviour.

Also here

• train_eval.py was a copy of the evaluation script pointed at the train split. The integrity notice asks contributors to develop on train and measure on validation, which only works if both are scored identically; they were not. It is now a thin wrapper over the same code, with per-error detail.
• benchmarks/datasets/ shadowed the datasets package under type checking, renamed to benchmarks/data/.
• benchmarks was never type-checked. files = ["src", "tests", "scripts"] omits it, while the release gate requires "strict mypy passes for source, tests, benchmarks, and scripts". It is added, and strict mypy now passes across it (it did not before: 4 errors).
• Roadmap drift. The current-position table ended at 0.7.0 | Next with 0.17.0 released; it now lists through 0.17.0. It also claimed a 99.36% coverage floor where pyproject.toml enforces 95.36%.
• 0.17.0 shipped with no changelog entry; one is written from its two commits.
• Three broken escapes rendered as literal backslashes: \ContextualIdDetector\ and \i4privacy/...\ (also missing its leading a).

249 passed, 1 skipped
ruff format, ruff check, mypy   clean

🤖 Generated with Claude Code

https://claude.ai/code/session_012JiX3zWeEC28kmy5KmAXvf

[mazzasaverio]

Rebased onto main at 0.19.0 and rescored, so the comparison is against current code rather than 0.17.0.

Version	Scoring	P	R	F1
0.17.0	as reported	0.9475	0.8910	0.9184
0.17.0	corrected, typed	0.8094	0.6536	0.7232
0.19.0	as reported	0.9479	0.8982	0.9224
0.19.0	corrected, typed	0.8097	0.6549	0.7241

Under corrected scoring, 0.18.0 and 0.19.0 together moved F1 by 0.0009: two more true positives out of 1611 annotations, which is inside run-to-run noise. The previously reported figures put the same interval at 0.9184 → 0.9224 and read it as a release-worthy gain.

Two observations that follow from the diff rather than from the numbers, offered as data for whoever reviews this:

1. 970f0f1 is titled "feat(ml): implement dynamic threshold calibration per entity type", but its diff is only the version bump in pyproject.toml. Between v0.17.0 and origin/main the sole source change in the whole repository is four lines in detectors/context.py. If the calibration work exists, it did not reach the branch.
2. CI on main is still failing the quality job for 2 files would be reformatted, now across four releases. fix: sanitize ONNX backend errors and restore the green formatting gate #57 fixes it; every PR here inherits the red gate until then.

Neither point changes this PR's content. They are the reason it exists: when the metric cannot distinguish a change from noise, a release can be attributed to code that is not there and nothing catches it.