benchmark(commercial_ehr): qualify the verifier on Epic's public FHIR sandbox

CAPABILITIES.md

@@ -145,6 +145,7 @@ improvising or retrying when something is off.
 | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` |
 | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md) |
 | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) |
+| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
 | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |

llms-full.txt

@@ -542,6 +542,7 @@ improvising or retrying when something is off.
 | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` |
 | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md) |
 | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ENSURE_RECORD.md) |
+| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/commercial_ehr/FINDINGS.md) |
 | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent <scripted|claude|cmd:...|py:...>` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) |
 | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) |
 | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |

packages/flow/benchmark/commercial_ehr/FINDINGS.md

@@ -12,6 +12,53 @@ what was read are at the end.
 | Oracle Health (code console; secure sandbox `fhir-ehr-code.cerner.com`, R4) | All four: Patient create and patch; Observation, AllergyIntolerance create and update; Appointment create and patch. The open sandbox is read-only | No AuditEvent, no `_history`. Provenance read, search and create | **No.** Benchmarking needs Oracle's prior written approval, test results belong to Oracle, and any public announcement about using the APIs is a material breach | **No-go** without Oracle's written approval |
 | athenahealth | not assessed | not assessed | not assessed: the developer portal and docs return 403 to automated reads | **Not assessed**; read the terms after signing in, if wanted |
 
+## Phase 2 on Epic: done, on a narrower read path
+
+The founder registered the app on 2026-10-29, and the sandbox accepted its
+token at once. What came of it is in [epic/](epic/README.md):
+
+- **Read path, probed** (`epic/probe.json`): search and read work, and the
+  server's `Date` header gives the window. There is no AuditEvent endpoint
+  (404), no `_history` (404) and no `meta` on records. `_lastUpdated` is
+  refused on Patient and silently ignored on Observation: a filter in 2099
+  still returned 266 rows, so a sweep there would be wrong, not just
+  incomplete. `_revinclude=Provenance:target` returns the latest author
+  and time; `_revinclude=AuditEvent:entity` returns nothing. A patient
+  search on one value alone matches nothing.
+- **What Epic does with a write:** a Patient create needs an SSN. Epic
+  answers `201` and **silently drops the phone number** in every format
+  tried, while keeping the email. It adds about ten elements of its own
+  (a second `official` name, race and ethnicity extensions,
+  `meta.security` PATRPT and UNCERTREL). A create of a person who already
+  exists, even under a new SSN or at another street, answers `201` with the
+  existing record's Location: `201 Created` doesn't mean created. An
+  allergy create needs `verificationStatus: unconfirmed`, lands (readable
+  by its Location), and **doesn't appear in the patient's allergy search**,
+  which still answers "Not on File". An Observation create needs an
+  Encounter, which the app has no scope to find.
+- **Verifier:** `openadapt_flow.verifier` with the `fhir` record check and
+  its existing `smart_backend` auth, contract `epic/register_patient.yaml`.
+  The sandbox exposed one verifier bug: the OperationOutcome Epic puts in an
+  empty search counted as a record, so a clean create read as a removed
+  record. It is fixed in `runtime/effects/fhir.py`, with tests.
+- **Fault suites** (`epic/RESULTS.md`): a **blind round** of 13 cases from
+  a separate author, frozen by sha256 before its one run, got 12 right with
+  no fault passed as verified. The miss was a false refusal of a clean save
+  beside a pre-existing look-alike, caused by the contract; the contract was
+  fixed to one `exact_new_set` effect, and both suites were rerun fully
+  right (not blind). The earlier non-blind rounds got 7 of 9 and 10 of 10.
+  The fix to the record check passed an adversarial review
+  (`epic/REVIEW.md`).
+
+What a claim from this may say: "a patient registration in Epic's public
+FHIR sandbox, read back through FHIR search and read". It can't say that
+nothing else changed: the frame is the task's search set, and who wrote the
+record isn't checked. `ensure_record` (found or created, with the audit
+trail) doesn't run on this sandbox, because its change feed needs AuditEvent
+or a working `_lastUpdated` sweep. Allergy writes can't be checked by
+search at all; a check would have to read by the Location the create
+returns, which trusts the agent's report of the write.
+
 ## Epic
 
 ### (a) Writes
@@ -54,13 +101,11 @@ every run that needs the feed ends `needs_review`. The record-level checks
 still work through search and read: the record exists once, holds every
 value, has no extra fields, and no duplicate exists.
 
-Phase 2 therefore has to qualify a narrower read path and name it:
-"the record and its duplicates, read through Epic's FHIR search; other writes
-through the agent's own client are seen only through `_revInclude` audit or
-provenance, where the sandbox returns it". That needs a contract
-`change_feed` mode the verifier doesn't have yet (per-resource `_revInclude`
-instead of a by-date AuditEvent search). Probe first, then decide whether to
-build it.
+Phase 2 confirmed this and checked the narrower path instead (see "Phase 2
+on Epic" above): the `fhir` record check of `openadapt_flow.verifier`,
+whose frame is the search set. A per-resource `_revinclude` change feed for
+`ensure_record` wasn't built: AuditEvent through `_revinclude` came back
+empty, and Provenance names only the latest author.
 
 The sandbox is shared with every Epic developer and Epic may wipe it at any
 time, so a run must use names and identifiers unique to the run, and a sweep
@@ -98,6 +143,12 @@ first is optional but cheap.
 
 ### (d) Registration (the founder does this; no agent creates accounts)
 
+Done on 2026-10-29. Two differences from the steps below: Epic checks a
+backend app's key against a JWK Set URL, not an uploaded certificate, so
+the public key is served as a JWKS from a secret gist; and Epic offers no
+Observation Update for vital signs or labs, and no AuditEvent API to backend
+apps, so neither was granted.
+
 1. Go to https://fhir.epic.com, choose **Sign Up**, and create an Epic on
    FHIR account with a work email. Confirm the email. Accept the Terms of
    Use when asked.
@@ -116,8 +167,8 @@ first is optional but cheap.
      AllergyIntolerance Read, Search, Create; Appointment Read, Search;
      Provenance Read. Add AuditEvent if the list offers it.
    - Default FHIR version: R4.
-   - Non-production JWT signing: upload `public_cert.pem` (or a JWK Set URL
-     if the form requires one; tell me and I'll set one up).
+   - Non-production JWT signing: a JWK Set URL serving the public key
+     (Epic requires one for backend apps).
    - Accept the terms checkbox, **Save**, then **Mark Ready for Sandbox**
      if that button is offered.
 4. Copy the **Non-Production Client ID** from the app's page.
@@ -128,11 +179,10 @@ first is optional but cheap.
    ```
    EPIC_SANDBOX_CLIENT_ID=<Non-Production Client ID>
    EPIC_SANDBOX_PRIVATE_KEY=/Users/abrichr/.config/openadapt/epic/private.pem
+   EPIC_SANDBOX_KID=<the kid in the JWKS>
+   EPIC_SANDBOX_ALG=RS384
    ```
 
-Then tell this session. Phase 2 starts with a read-only probe
-(`probe.py` like Medplum's), then the write probe, then the contract.
-
 ## Oracle Health
 
 ### (a) Writes

packages/flow/benchmark/commercial_ehr/epic/MINUTES.md

@@ -0,0 +1,52 @@
+# Who wrote what, and how long it took: Epic against Medplum
+
+Agent wall-clock on 2026-10-29, from command timestamps in the session
+(12:41 to 13:05 EDT), rounded. The person's part was the registration
+(FINDINGS.md, section (d)), which took them a few minutes and isn't counted
+below.
+
+## The contract
+
+`register_patient.yaml` has 46 lines that aren't comments, all written by
+hand. The #1520 generator (`generate-parts`) wasn't used: it learns field
+paths from demonstrations of an app's form, and Epic's sandbox has no form
+to demonstrate, only the FHIR API. The field paths are the US Core Patient
+paths, the same as in Medplum's generated contract, so writing them took
+minutes.
+
+| Part | What it took | Minutes |
+| --- | --- | --- |
+| Verifier block: base URL, SMART backend (client ID, kid, RS384 key from `.env`), resource type | the product's `smart_backend` config, unchanged | 2 |
+| Search: family name and birth date (Epic refuses a search on one value) | one probe, two lines | 1 |
+| Field paths and effects: name, birth date, sex, street, city, state, postal code, email | copied from the US Core paths | 3 |
+| Phone variant | one line each in a second file | 1 |
+| **Contract total** | | **about 7** |
+
+## The read path
+
+| Finding | What it took | Minutes |
+| --- | --- | --- |
+| No AuditEvent, no `_history`, no `meta`; `_lastUpdated` refused on Patient and ignored on Observation | read-only probe, 14 calls | 3 |
+| A create needs an SSN; Epic drops the phone and adds about ten elements of its own | write probe | 4 |
+| A create of a person who exists answers 201 with the old Location | write probe | 2 |
+| An allergy create lands but doesn't show in the patient's allergy search | write probe | 4 |
+| The empty-search OperationOutcome counted as a record | a product fix (one branch) and two tests | 6 |
+| **Read-path qualification total** | | **about 19** |
+
+The suite and the two rounds took about 10 more minutes, including the
+case rewrites in RESULTS.md.
+
+## Against Medplum
+
+| | Medplum (FHIR + AuditEvent + `_history`) | Epic sandbox (FHIR search and read) |
+| --- | --- | --- |
+| Contract lines written by hand | 16 (14 generated) | 46 (no generator: no form to demonstrate) |
+| Agent minutes on contract and read path | about 43 | about 26 |
+| What the frame covers | every write the audit trail logs, any resource type | the search set only: patients with the task's family name and birth date |
+| Who wrote a change | the AuditEvent agent | not checked; Provenance names an author only for the latest change |
+| Fault rounds | 3 blind rounds by separate authors | 2 rounds by the contract's author (not blind) |
+
+Epic took fewer minutes because there was less to qualify, not because it
+is better: the read path that exists is narrower. Any claim from this run
+names that path: "the patient record, read back through Epic's FHIR search
+and read".

packages/flow/benchmark/commercial_ehr/epic/README.md

@@ -0,0 +1,40 @@
+# Registering a patient in Epic's public FHIR sandbox, checked through FHIR only
+
+An agent registers one new patient in Epic's public sandbox (open.epic,
+R4). The verifier (`openadapt_flow.verifier`, `begin_write` and
+`verify_write`) reads the sandbox before and after the write with its own
+SMART backend token and judges the write against `register_patient.yaml`:
+`CONFIRMED`, `REFUTED` or `UNCERTAIN`. It reads Patient searches and
+records only. Epic's sandbox has no AuditEvent search and no `_history`, so
+the frame is the search set: every patient with the task's family name and
+birth date. [FINDINGS.md](../FINDINGS.md) has the terms and the read path.
+
+| File | What it is |
+| --- | --- |
+| `register_patient.yaml`, `register_patient_phone.yaml` | the contracts, without and with the task's phone number |
+| `client.py` | the backend OAuth client (an RS384-signed JWT for a token) |
+| `probe.py`, `probe.json` | the live checks of what Epic's read path shows and what it does with a write, and their results on 2026-10-29 |
+| `suite.py` | the write-fault suite: ten built-in cases, or a frozen suite file checked by its sha256; each case a run-unique synthetic patient |
+| [RESULTS.md](RESULTS.md), `round1.json`, `round1_followups.json`, `round2.json`, `round3.json`, `blind_summary.json` | the rounds, the reruns between them, and the blind round's counts |
+| [REVIEW.md](REVIEW.md) | the adversarial review of the verifier fix |
+| [MINUTES.md](MINUTES.md) | who wrote what, and how long it took |
+
+## Running it
+
+An Epic on FHIR account and a Backend Systems app with the R4 Patient,
+AllergyIntolerance and Observation (vital signs) read and create APIs and
+Provenance read. FINDINGS.md has the registration steps. The credentials
+live in `.env`, never here:
+
+```bash
+cd packages/flow
+set -a; source /path/to/.env; set +a
+PYTHONPATH=. python -m benchmark.commercial_ehr.epic.probe --out probe.json
+PYTHONPATH=. python -m benchmark.commercial_ehr.epic.suite --out results.json
+```
+
+The sandbox is shared with every Epic developer, Epic may wipe it at any
+time, and the app can't delete. Each run writes about a dozen synthetic
+patients with family names unique to the run. The SSNs that Epic's create
+requires come from the never-issued 000 area. Don't loop or schedule the
+suite: Epic's terms rule out load and performance testing on the sandbox.