fix(detectors): stop the context detector firing on ordinary prose - #59
Open
mazzasaverio wants to merge 2 commits into
Open
fix(detectors): stop the context detector firing on ordinary prose#59mazzasaverio wants to merge 2 commits into
mazzasaverio wants to merge 2 commits into
Conversation
Three release-gate defects, all reproducible on main. The ONNX backend interpolated the originating exception message into BackendExecutionError. The release gate requires that exceptions do not expose matched values, and PR ma2za#40 established the same invariant for adapters; tokenizer and runtime messages can quote the input they failed on. The message is now fixed text and the cause is dropped, matching how every other adapter and backend in the tree raises. Detection also scanned every character with unicodedata.category on every block, building two N-sized lists even when nothing was stripped. No ASCII character has category Cf, so pure ASCII text now takes a single C-level scan and keeps the source offsets untouched, and non-ASCII text only pays for the rebuild when a format character is actually present. This is 13% of end-to-end processing time on a 64 KiB block. Finally, `uv run ruff format --check .` fails on main: the 0.16.0 OCR commit landed unformatted, so CI has been red across 0.16.0 and 0.17.0 even though both shipped. The two files are reformatted here so the gate passes again. uv.lock still recorded 0.16.0 and is refreshed to 0.17.0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012JiX3zWeEC28kmy5KmAXvf
The contextual identifier patterns were tuned against the ai4privacy
evaluation corpus rather than against how documents are actually written,
which is the practice the benchmark harness's own integrity notice rules
out. Two problems followed from it.
A label was treated as sufficient evidence on its own, with no requirement
that the captured value look like an identifier. On main:
"Please provide identification number before Tuesday"
-> "before" NATIONAL_ID
"The applicant Jonathan submitted forms."
-> "Jonathan" NATIONAL_ID
"We approved a budget of 1500000000000 lire."
-> "1500000000000" PAYMENT_CARD
"Your school 100200 is nearby."
-> "100200" LOCATION
The 0.15.0 digit constraint was added to two patterns but not to the
national-identity one, which still matched any six-to-fifteen character run.
Every capture now requires at least one digit.
Several triggers were also not labels at all. "budget of", "contribution of",
"school", "office at", "singapore", "sending your", "proof like a", "cung
cấp" ("provide") and "número de impresos" ("number of printed forms") do not
precede an identifier in any document; they precede one in particular rows of
one dataset. They are removed, along with the bare "id", "serial",
"reference" and "applicant" triggers that matched without any number marker.
The genuine multilingual labels stay, and "reference" and friends still match
when written as "reference number".
Ten labelled identifiers across English, Vietnamese and Chinese are covered
as regression tests alongside the nine prose cases, so the loosening cannot
come back unnoticed.
This will lower the reported benchmark precision figure, because some of
these matches were scoring as true positives against the corpus they were
derived from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012JiX3zWeEC28kmy5KmAXvf
mazzasaverio
force-pushed
the
fix/context-detector-overfitting
branch
from
August 30, 2026 05:33
87317d0 to
b719f9e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The contextual identifier patterns were tuned against the ai4privacy evaluation corpus rather than against how documents are written, which is the practice
benchmarks/evaluate_quality.py's own integrity notice rules out:Two problems follow from it.
A label was treated as sufficient evidence
Nothing required the captured value to look like an identifier, so an ordinary English word sitting where the identifier belongs was reported as personal data. On
main:0.15.0's changelog says the digit constraint was added to the
NATIONAL_IDcontextual fallback, but it reached only two of the patterns; the national-identity one still matched any six-to-fifteen character run, letters included. Every capture now requires at least one digit.Several triggers were not labels at all
budget of,contribution of,school,office at,singapore,sending your,proof like a,cung cấp("provide") andnúmero de impresos("number of printed forms") do not precede an identifier in any document. They precede one in particular rows of one dataset. They are removed, along with the bareid,serial,reference,identifierandapplicanttriggers that matched with no number marker at all.The genuine multilingual labels stay —
mã số thuế,căn cước,número de identificación,身份证号,护照号,paspor,nomor SIM— and the reference-number family still matches when written as a label:reference number,ticket number,serial no.Verified both directions
Nine prose cases that must not match and ten labelled identifiers across English, Vietnamese and Chinese that must, all as parametrized regression tests:
Expect the benchmark number to move
The reported precision figure will fall, because some of these matches were scoring as true positives against the very corpus they were derived from. A detector that fires on "the applicant Jonathan" is not more precise in the field; it scores better on one holdout slice. #59 addresses the harness those numbers come from.
🤖 Generated with Claude Code
https://claude.ai/code/session_012JiX3zWeEC28kmy5KmAXvf