Skip to content

problems_only switch policy: input problems leave it on, a person re-arms with one signed tap - #1517

Merged
abrichr merged 8 commits into
mainfrom
fix/switch-problem-classes
Oct 25, 2026
Merged

abrichr merged 8 commits into
mainfrom
fix/switch-problem-classes

Conversation

@abrichr

@abrichr abrichr commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

This PR implements the coordinator's decision on the problems_only switch policy: classify every problem, and let a person re-arm only with a signed tap. It is stacked on OpenAdaptAI/OpenAdapt#1514.

Why: in the measured soak (OpenAdaptAI/OpenAdapt#1513), one caught problem turned the switch off for the rest of the run. Every later save then needed an approval: 25 of 125.5 operator minutes.

1. Which problems turn it off (post.problem_class, classify_decision)

Item-input problems leave the switch on. The item goes to a person and doesn't count toward too_many_problems. These are problems caught before anything was written:

Program and environment problems turn it off at once. For example:

  • a wrong write the record check caught;
  • a failed identity check;
  • a postcondition that failed after delivery;
  • a changed screen;
  • the wrong signed-in user;
  • any unnamed cause.

Any outcome that doesn't consistently read "nothing was written" is a program problem (fail closed). A switch-off the gate, R2 or the engine already named is never classed down to an input problem. Gate pre-check failures map to closed off reasons: signed_in_user → environment_changed, right_record and same_screens → screen_changed, record_check_ready → record_check_lost. too_many_problems stays as a backstop.

2. A person re-arms with one signed tap (switch.rearm_request, sign_rearm, rearm)

  • Never automatic, under any profile.
  • Offered only after 10 consecutive verified items since the switch went off. These are no_person or approved items; any other finished item resets the count.
  • What the person's approver key signs: the off event and its reason, the recorded cause, and those 10 run refs.
  • What the log records: a rearmed event with who signed, the key, and what they saw. The field is left out of every other event, so existing log lines and hashes are unchanged.
  • Who honours it: a gate that trusts the signing key. By default, and deliberately, those are the approvers registered in the JIT key directory's principal registry. A person registers each at a terminal (jit register --role approver, fix: a person's approval needs a registered approver key's signature #1486), and the registry refuses agent and model names. ApprovalGate(pinned_approver_keys=...) names keys explicitly instead. Keys may be hex (the registry's form) or base64.
  • Binding: a re-arm's approver_id must equal its key_id (the key's registered name).
  • Only a verified re-arm ends an off period. Every reader checks the re-arm again against the log before it, so a forged, tampered, replayed or early re-arm appended to the log is ignored and doesn't reset the count.
  • Signing outside Python: rearm_to_sign returns the exact bytes, and rearm_evidence wraps a signature made elsewhere. Desktop's Touch ID card (feat(desktop): 0.24.0, the Touch ID approver key and one-tap stop review #1519) uses these.
  • Before fix: a person's approval needs a registered approver key's signature #1486 lands, main's registry has no approvers, so no re-arm counts.

3. A bug the soak exposed

_unapproved_delivered_saves classified saves without the certifying policy, as the gate's own save check did before #1455. On OpenEMR, "Create New Patient" (certified as not saving) counted as an unapproved delivered save. So after the switch turned off, every approved, verified item was logged as a problem (save_not_approved), and it could never have counted toward a re-arm.

Tests

  • New test_approval_mode_switch_policy.py (23). Covers:
    • review cases: an explicit switch-off with an input-class halt stays off; a duplicate stop counts as input only from its structured outcome, and an unreadable one turns it off; a replayed old re-arm after a second switch-off; a re-arm for another switch log; a forged re-arm, then a real one; a mismatched approver; the gate reading on after a valid re-arm, and a gate that doesn't trust the key reading off;
    • registry keys in hex; an unreadable registry gives no keys; signing with a key that never leaves its keychain;
    • an invalid date, a missing value and a duplicate candidate leave the switch on;
    • a caught wrong write turns it off (uncertain_result);
    • an unnamed stop before any effect turns it off;
    • a duplicate stop after a delivered save, or an inconsistent outcome, is a program problem;
    • re-arm: signed after 10 verified items turns it on;
    • refused before 10, or after a problem resets the count;
    • refused unsigned, with an unpinned key, tampered, or forged;
    • fifteen clean items alone change nothing;
    • a later problem turns it off again;
    • a forged rearmed event appended to the log is ignored.
  • Old-policy tests updated (4). They expected a counted problem without a switch-off, or three in a row before one.
  • Locally: every test that touches the switch log or approval modes passed (797 in 32 files); after the review fixes, the approval-mode suites pass (360), and ruff and mypy pass.

🤖 Generated with Claude Code

abrichr and others added 3 commits October 24, 2026 19:46
…program problems turn it off, a person re-arms with one signed tap

- post.problem_class sorts every problem. Item-input problems caught before
  anything was written (a missing or invalid value refused before the first
  step, an input item check, a real duplicate candidate) send the item to a
  person and leave the switch on. Program and environment problems, and any
  unnamed cause, turn it off at once.
- A switched-off switch never re-arms itself. After 10 consecutive verified
  items, a person's approver key signs a re-arm over the off event, its
  cause and those items; the log records it, and a gate honours it only with
  the key pinned. A forged, tampered or early re-arm is ignored.
- The gate's count of unapproved delivered saves now uses the certifying
  policy, as its own save check does. A step the certified qualification
  classified as not saving made every approved, verified item a problem.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arm_to_sign, rearm_evidence)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr added a commit that referenced this pull request Oct 25, 2026
- The re-arm card shows Flow's re-arm request: why the switch went off,
  when, and the 10 items saved and checked in a row since, named. Its id
  covers the request, so a re-arm signs only what the person saw.
- Answering it is one Touch ID over Flow's own bytes (rearm_to_sign); Flow
  verifies against the approvers registry, never a key Desktop supplies.
  Desktop's switch view honours a re-arm by a registered approver.
- Feature-detected: the card appears only with a Flow that has signed
  re-arm (#1517). Checked against #1517's real switch log.
- A locked Mac no longer reads as having no Secure Enclave: the check isn't
  remembered, and the refusal says to unlock the Mac.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr and others added 3 commits October 24, 2026 20:21
…s by default

The run gate reads the approver keys registered in the JIT key directory's
principal registry (hex, as jit register --role approver writes them), so a
re-arm signed with Desktop's approver key counts at run time with no extra
configuration. pinned_approver_keys still names keys explicitly. An
unreadable registry, or one with no approvers, gives no keys, and a
switched-off switch stays off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- An explicit switch-off from the gate, R2 or the engine is never classed
  down to an input problem.
- A duplicate stop counts as an input problem only from the duplicate
  check's structured outcome (StepResult.duplicate_check == 'candidate',
  set by the record-channel check, #1516), never from the error text. An
  unreadable list or channel turns the switch off (record_check_lost); a
  duplicate stop that doesn't say is a program problem.
- A re-arm's approver must be its key's registered name.
- Only a re-arm that verifies ends an off period, so a forged event can't
  hide a switch-off or reset the count.
- Trusting the registered approvers by default is documented as deliberate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uplicate candidates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr added a commit that referenced this pull request Oct 25, 2026
StepResult.duplicate_check is set whenever no_duplicate_candidate fails:
exact or close (a record matched) and candidate (a listed row matched) are
about the item; unreadable (the record channel or the screen list could not
answer, or a record-pinned check had no channel) is about the program. None
otherwise and then omitted, so existing reports serialize unchanged. #1517
classes the problems_only switch from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the pinned root

#1486 moved approvers out of principals.json into a signed roster. The
gate's default approver keys are now the roster entries the run's pinned
root vouches for (approver_keys_b64, pinned_root), imported by name so this
works before and after #1486 merges; with no roster, no pinned root, or a
roster that doesn't verify, no re-arm counts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from fix/relabel-repair-duplicate-list to main October 25, 2026 05:41
@abrichr
abrichr enabled auto-merge October 25, 2026 05:41
@abrichr
abrichr merged commit eb1e285 into main Oct 25, 2026
40 checks passed
@abrichr
abrichr deleted the fix/switch-problem-classes branch October 25, 2026 07:29
abrichr added a commit that referenced this pull request Nov 9, 2026
…nstead of the screen list (#1516)

* duplicate check: ask the system of record for the patient (FHIR, REST; SQL fallback)

no_duplicate_candidate stopped every new patient who shares a last name or a
date of birth with any chart. When the deployment's effects read a FHIR
server or OpenEMR's patient API, the check now searches the record channel
(every page, fail closed) and stops only on an exact or close match. The
on-screen list stays the fallback and runtime.duplicate_check: screen keeps
it. Offline on the soak's seed-7 stream: false stops per 100 items 44.8
(screen) against 0.4 (record), the same 12 of 110 true duplicates missed by
both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: nicknames, swapped names, prior names, contact tie-breakers

The first version missed every married name with a mistyped day of birth;
in each, the phone, street and email matched the chart. The record check now
also searches by first name, phone and email, knows common nicknames, swapped
first and last names and the chart's birth name, and uses shared contact
details to tie a mistyped date of birth to a chart. Every new match stops for
a person.

Extended seed-7 stream (622 items, 169 true duplicates): misses 60 -> 24, all
24 a name change plus a move that shares nothing with the chart; false stops
unchanged at 2 (0.3 per 100 items). Screen: 252 false stops, 30 misses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test: the record duplicate check's halt keeps the no_duplicate_candidate prefix

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: an initial matches its first name; plant production-lifecycle's six cases

All six cases production-lifecycle named stop as close matches: a married
name with a new address, a mistyped day, a second surname, an initial, a
crossed name, and both names misspelled with the same contact. The initial
was the one gap. Extended stream (676 items, 223 true duplicates): false
stops 2 (0.3 per 100 items), misses 24, all name change plus move.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs: duplicate check, the exact test-set size and misses by kind

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: one source for every Replayer; a program can pin the record check

The record source was set only in deployment.build_replayer, so
qualification (qualify/campaign.py), the lifecycle proof's run, execute and
resume built Replayers that silently used the on-screen list. Replayer now
resolves the source itself from the effect verifier it is given
(resolve_duplicate_source), so every path that reads the same system of
record uses the same check. The postcondition's candidate_source: record
pins the record check into the qualified program; without a channel the
step fails closed instead of reading the screen. The soak's OpenEMR program
change pins it.

tests/test_duplicate_source_paths.py scans every Replayer(...) construction:
none picks a source of its own, the governed paths pass their verifier, the
same verifier gives the same source through build_replayer and directly, and
a pinned program fails closed without a channel. RestPatientSource and
SqlPatientSource are labeled OpenEMR adapters; FhirPatientSource generic R4.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: review fixes, one resolution for every path, REST opt-in

CI: resolution no longer reads attributes a verifier double lacks (unit
shards), the CAPABILITIES row states what was measured (copy test), and the
public artifact inventory is regenerated (lint).

Fail-closed reads: a REST body with any error field or no data list, a
partial record birth date (now no date, so a name match is close), a record
with no id, and a next-page link to another host are all unreadable instead
of a silent none.

One resolution: the deployment switch is gone; the program's
candidate_source (part of the qualified program) and the run's own verifier
decide on every path, and a qualify hook can't set the source. FHIR is the
only automatic channel; OpenEMR's REST adapter needs candidate_source:
record. candidate_record_fields maps a program's parameter names. The SQL
adapter uses MySQL %s parameters and a bound wildcard.

Recall: a nickname or a last-name typo with a mistyped date of birth now
stops. Extended stream (694 items, 241 duplicates): misses 24 (all name
change plus move), false stops 2 (0.3 per 100); screen 42 and 252.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: the halted step carries a structured outcome

StepResult.duplicate_check is set whenever no_duplicate_candidate fails:
exact or close (a record matched) and candidate (a listed row matched) are
about the item; unreadable (the record channel or the screen list could not
answer, or a record-pinned check had no channel) is about the program. None
otherwise and then omitted, so existing reports serialize unchanged. #1517
classes the problems_only switch from it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: live OpenEMR probe and what it found

benchmark/dup_check/live.py seeds synthetic charts on the pinned fixture and
probes OpenEMR's REST and FHIR patient search. REST honors _limit and gives
links.next, but on OpenEMR's configured site address, which the check refuses
as another origin (fail closed): REST stays opt-in. FHIR gives no next link
and a total equal to the rows returned; without _count it returned every row
(120 tested). The check decided 6 of 6 referrals right through FHIR and REST
at 500 a page. Raw output is private (openadapt-corpus#18).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: a program can choose FHIR beside an OpenEMR REST record check

candidate_channel: fhir with a RestRecordVerifier on .../apis/default/api/patient
searches OpenEMR's FHIR API on the same server (.../apis/default/fhir) with
the record check's bearer token; rest uses only the REST adapter. Live, REST's
next-page links carry OpenEMR's configured site address, so FHIR is the
channel that answers completely there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: per-item results out of the tree

The offline measurement's per-item rows (about 8.5k lines) move to the
private openadapt-corpus; the README keeps the per-kind tables. measure.py
now requires --out so it never writes into the repository.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* duplicate check: live FHIR rates harness (runs on the fixture lease)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* CAPABILITIES: no duplicate-check rates until the live FHIR re-run

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Duplicate check docs: exact matches stop, close matches go to a person, heuristics miss some

* Duplicate check: zero dates, names as typed, FHIR cut-short pages, OpenEMR adapter

A record date that is no real date (0000-00-00, 1980-02-30) counts as no
date, so a name match is close. Name searches also send the name as typed,
since channels compare the stored text. A full FHIR page with no next link
is unreadable. OpenEMR's REST and SQL sources and the REST-verifier mapping
move to runtime/checks/openemr_patient.py; the core is generic FHIR.

* dup_check README: table compares stored name text

* classify: a record date that is no real date counts as none, whatever the source

* Duplicate check: app knowledge lives in data, not in the engine

The engine generality check refused runtime/checks/openemr_patient.py: an
OpenEMR adapter in openadapt_flow. Each app's search is now a data file,
app_profiles/record_search/<app>.yaml: the REST path marker, where its
FHIR API sits beside the REST API, the REST rows key, next-page link,
query parameters and field names, and the SQL table and columns.
runtime/checks/app_sources.py follows that file. OpenEMR's file holds what
the old module hard-coded, so the searches are the same.

A REST record check is matched to the one app whose marker its records
path contains; none or two matches give no source, so a program pinned to
the record check fails closed as before. The SQL table and every column
must be plain identifiers, and values stay bound.

* Duplicate check: match first names by their parts, keep every given name and script

A first name whose parts are all among the other's (Mary and Mary Ann,
Maria Jose and Maria) is now a close match when the last name and date
of birth agree; it was none. FHIR keeps given[1:] and REST/SQL read
mname, so a chart's whole given name is compared. Names keep letters of
every script, so different Cyrillic, Greek or CJK names no longer
normalize to the same empty string, and Soundex applies only to Latin
names. A referral phone matches any phone on the chart, not only home.

* Duplicate check: keep every given name of a FHIR prior name

A FHIR old or maiden HumanName kept only given[0], so a chart formerly
Mary Ann Ashford was none for Ann Ashford while the table, REST and SQL
channels said close. Every HumanName now keeps all its given parts, and a
prior name is compared with the same first-and-middle rule as the
current one.

* fix(benchmark): refuse abbreviated options in the dup_check CLIs

The CLI census requires allow_abbrev=False on every parser.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant