Skip to content

fix: decide separated action text by shape, and label identifiers by recognizer - #19

Merged
abrichr merged 1 commit into
mainfrom
fix/scrub-dict-separated-and-entity-labels
Aug 28, 2026
Merged

fix: decide separated action text by shape, and label identifiers by recognizer#19
abrichr merged 1 commit into
mainfrom
fix/scrub-dict-separated-and-entity-labels

Conversation

@abrichr

@abrichr abrichr commented Aug 28, 2026

Copy link
Copy Markdown
Member

Two defects, both reproduced against the published 1.0.3 on macOS.

1. scrub_dict mangled a plain text value

scrub_dict({"text": "Email: john@example.com"}, scrubber)
# before: {'text': '<-P-E-R-S-O-N->-:- -<-E-M-A-I-L-_-A-D-D-R-E-S-S->'}
# after:  {'text': '<PERSON>: <EMAIL_ADDRESS>'}

text is in SCRUB_KEYS_HTML, so _scrub_text_item passed is_separated=True
for it. Inside scrub_text the split and the join were not symmetric: the split
on ACTION_TEXT_SEP is a no-op on a value that holds no separator, but the join
ran anyway and put a separator between every character of the result.

The design question: opt-in per call, or keyed off the field name?

Neither. It is keyed off the value's shape, and the key name only grants
permission.

A per-call flag pushes the decision onto a caller who often cannot make it.
scrub_dict recurses through an arbitrary nested structure, so one flag has to
cover every value at every depth, and a GUI element tree legitimately mixes
recorded keystrokes with element labels under the same key name. The shape test,
by contrast, is exact for the encoding in question: action text is
ACTION_TEXT_SEP.join(chars), so every separated chunk is exactly one
character. follow-up and 555-123-4567 fail that test; j-o-h-n passes it.

Applying the test in the wrong direction costs nothing in either direction:

  • A genuine key sequence still gets reassembled before analysis, so PII that is
    invisible in the split form is still found. This is the direction that would
    leak, and it is preserved.
  • A value that really is all single-character chunks round-trips to itself, so
    a false positive is a no-op.

separated_keys is still exposed on scrub_dict / scrub_list_dicts, backed
by the new PrivacyConfig.SCRUB_KEYS_SEPARATED, so a caller who wants it off
entirely passes separated_keys=[] and one who uses a different field name
passes their own. The hard-coded ("text", "canonical_text") tuple is gone.

Callers checked

No repository in the workspace calls scrub_dict, scrub_list_dicts or
DictScrubber from this package at all.

repo how it uses openadapt-privacy
openadapt-flow PresidioScrubbingProvider().scrub_text(text) in privacy.py, hosted.py, sanitized_artifact.py, compiler/compile.py, console/halt_detail.py, scripts/check_bundle_phi.py — always the one-argument form, never is_separated. Its tests inject fakes, so no test pins a placeholder name from the real analyzer.
openadapt-desktop engine/scrubber.py calls self._provider.scrub_text(text). Same one-argument form.
openadapt-capture declares the dependency in pyproject.toml, calls nothing.
openadapt-cloud, -tray, -evals, -ml no import.
OpenAdapt (legacy) has its own vendored legacy/openadapt/privacy/base.py. Separate code, not this package.

So the action-text path this behaviour existed for is not live in any current
consumer, and the shape gate preserves it anyway for the one that revives it.

2. Entity labels named the wrong type

'Card 4111111111111111 on file'           -> <DATE_TIME>       now <CREDIT_CARD>
'My bank account number is 635526789012.' -> <DATE_TIME>       now <US_BANK_NUMBER>
'Last login from 192.168.10.22.'          -> <DATE_TIME>       now <IP_ADDRESS>
'Health plan member ID ZXQ-443-991.'      -> <PERSON>          now <MEDICAL_RECORD_NUMBER>

Root cause is not recognizer ordering or a threshold. analyzer.analyze()
returns all four candidates for the first case, CREDIT_CARD among them at
score 1.0:

CREDIT_CARD       [5,21] 1.00 '4111111111111111'
DATE_TIME         [0,21] 0.85 'Card 4111111111111111'
US_BANK_NUMBER    [5,21] 0.05
US_DRIVER_LICENSE [5,21] 0.01

Presidio's anonymizer resolves the overlap by span before score, so the
wider spaCy DATE_TIME won and took the word "Card" with it. A statistical NER
span outranking a Luhn-validated pattern match is the inversion.

_prefer_deterministic_identifier_labels drops the statistical result where a
pattern recognizer scoring at least 0.4 overlaps it. The 0.4 floor keeps the
0.01–0.05 context-free digit-run guesses from taking a label away from NER.

Redaction coverage cannot shrink. The statistical result is dropped only when
the pattern matches cover every digit its span held. Presidio's driver
licence recognizer matches only the leading A123 of A123-456-789-012, so a
wider span over the rest survives and the trailing digits stay redacted — there
is a unit test for exactly that.

One case is not a code defect

'Card: 4532-1234-5678-9012' still returns <DATE_TIME>. That number fails its
Luhn check, so Presidio's card recognizer invalidates it and CREDIT_CARD never
enters the candidate list. The fixture was wrong, not the analyzer. The recall
suite now uses 4532-1234-5678-9014, and the old number has its own test
pinning the documented limit.

Not fixed, reported

A123-456-789-012 still scrubs to <US_DRIVER_LICENSE>-456-789-012. That is a
span problem in Presidio's upstream recognizer, not a label problem, and
tightening it needs a new pattern with its own false-positive budget. Separate
work.

Tests

tests/test_phi_recall.py asserted only that the identifier had disappeared,
which is why a card number reading <DATE_TIME> passed for months. Every case
now pins the entity type too. Both new tests were watched failing first:

FAILED tests/test_phi_recall.py::test_synthetic_phi_entity_labels_are_correct
  5/26 cases carry the wrong entity type: [('credit_card', 'CREDIT_CARD', '<DATE_TIME> is on file.'),
  ('credit_card', 'CREDIT_CARD', '<DATE_TIME> on file'), ('bank_account', 'US_BANK_NUMBER',
  'My bank account number is <DATE_TIME>.'), ('ip', 'IP_ADDRESS', 'Last login from <DATE_TIME>.'),
  ('member_id', 'MEDICAL_RECORD_NUMBER', 'Health plan member <PERSON>.')]
FAILED tests/test_separated_action_text.py::test_plain_prose_under_the_text_key_is_not_character_separated
FAILED tests/test_separated_action_text.py::test_hyphenated_prose_under_the_text_key_keeps_its_words
FAILED tests/test_separated_action_text.py::test_separated_handling_can_be_turned_off_per_call
FAILED tests/test_separated_action_text.py::test_a_phone_number_under_the_text_key_survives_as_a_phone_number
5 failed, 13 passed

118 pass after the fix. No existing test was weakened; the recall gate still
requires 26 of 26.

Docs

#18 documented both defects honestly and merged an hour before this branch.
This PR is rebased on it and rewrites the two paragraphs the fix invalidates:
the Card ... -> <DATE_TIME> example and the "one caveat in 1.0.3" note about
the text key. The Luhn limit replaces them, with a measured example of each
side. CLAUDE.md's entity table also claimed 4532-1234-5678-9012 produces
<CREDIT_CARD>; it now uses a valid number.

Released

Shipped as v1.0.4 and verified against the wheel installed from PyPI:

>>> scrub_dict({'text': 'Email: john@example.com'}, s)
{'text': '<PERSON>: <EMAIL_ADDRESS>'}
>>> s.scrub_text('Card 4111111111111111 on file')
'Card <CREDIT_CARD> on file'
>>> s.scrub_text('My bank account number is 635526789012.')
'My bank account number is <US_BANK_NUMBER>.'

The release workflow break reported earlier today (type object 'Actor' has no attribute 'name_email_regex' inside python-semantic-release) did not
reproduce: the post-merge run tagged, released and published without
intervention.

…recognizer

Two defects, both reproduced against 1.0.3.

scrub_dict mangled a plain "text" value. "text" is in SCRUB_KEYS_HTML, so
_scrub_text_item passed is_separated=True for it, and scrub_text re-joined the
anonymized result with ACTION_TEXT_SEP whether or not the input had ever been
split. A sentence came back one character per hyphen:

    scrub_dict({"text": "Email: john@example.com"}, scrubber)
    {'text': '<-P-E-R-S-O-N->-:- -<-E-M-A-I-L-_-A-D-D-R-E-S-S->'}

The split and the join are now symmetric: is_separated permits key-sequence
handling, and _is_separated_form decides whether to apply it by looking at the
value. Action text is exactly one typed character per separator, so "j-o-h-n"
is reassembled and re-separated as before, while "follow-up", "555-123-4567"
and ordinary prose are scrubbed as prose. Which keys may carry action text is
now PrivacyConfig.SCRUB_KEYS_SEPARATED instead of a hard-coded tuple, and
scrub_dict/scrub_list_dicts take separated_keys to set it per call.

Entity labels named the wrong type, so policy keyed on them was unusable.
Presidio's anonymizer settles overlapping spans by span before score, so a
spaCy DATE_TIME at 0.85 covering "Card 4111111111111111" replaced a
Luhn-checked CREDIT_CARD at 1.0. Same for US_BANK_NUMBER, IP_ADDRESS, and a
member ID that came back as PERSON. Nothing leaked, but nothing could be
routed on either.

_prefer_deterministic_identifier_labels drops the statistical result where a
pattern recognizer scoring at least 0.4 claims the span. It drops it only when
the pattern matches cover every digit the statistical span held, so relabelling
can never uncover part of an identifier: Presidio matches only the leading
"A123" of "A123-456-789-012", and a wider span over the rest survives.

A card number that fails the Luhn check still gets no CREDIT_CARD label,
because no recognizer validates it. That is now a documented, tested limit
rather than a surprise, and the README example uses a valid number.

test_phi_recall.py asserted only that the identifier had disappeared, which is
why this went unnoticed. Every case now pins the entity type as well.
@abrichr
abrichr merged commit c8a8cfd into main Aug 28, 2026
4 checks passed
@abrichr
abrichr deleted the fix/scrub-dict-separated-and-entity-labels branch August 28, 2026 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant