diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 066e37ede..824739ed2 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -145,6 +145,7 @@ improvising or retrying when something is off. | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md) | | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](packages/flow/docs/ENSURE_RECORD.md) | +| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](packages/flow/benchmark/commercial_ehr/FINDINGS.md) | | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent ` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](packages/flow/benchmark/fhir_medplum/public/RESULTS.md) | | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | diff --git a/llms-full.txt b/llms-full.txt index f8b474d0a..b5e9d2c92 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -542,6 +542,7 @@ improvising or retrying when something is off. | One check-level scale, 0 to 5, from "unchecked" to "counterparty confirmed" | Released | Unit tests | `openadapt_flow/check_level.py` | | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md) | | Judge an "ensure the record exists" write (found, created once with every value, or needs review) through a FHIR server's read path only: searches, AuditEvent and `_history`, never the database; the coverage claim names that read path | On main ([#1640](https://github.com/OpenAdaptAI/OpenAdapt/pull/1640)) | Real app (Medplum 5.2.8; three blind fault suites: 17/18, 17/20, 19/20; no fault passed in the last two) | `openadapt_flow/ensure_record.py`, [ENSURE_RECORD.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ENSURE_RECORD.md) | +| Check a patient registration on a commercial EHR through its public FHIR sandbox: Epic (open.epic, R4), read through FHIR search and read with SMART backend auth; the frame is the search set, since the sandbox has no AuditEvent or `_history` | On main ([#1656](https://github.com/OpenAdaptAI/OpenAdapt/pull/1656)) | Real app (Epic public sandbox, August 2026 release; blind round 12/13 with none passed as verified, the miss a false refusal fixed in the contract; caught Epic dropping the phone on a 201 create) | `benchmark/commercial_ehr/`, [FINDINGS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/commercial_ehr/FINDINGS.md) | | A public benchmark a vendor reruns with their own agent: `openadapt bench medplum --agent ` runs 37 written-for-it cases (Patient, AllergyIntolerance, Appointment) against a local Medplum through a fault proxy, and scores the agent's claim, a read-back and OpenAdapt's verifier against an end-state oracle | On main ([#1652](https://github.com/OpenAdaptAI/OpenAdapt/pull/1652)) | Real app (Medplum 5.2.8; Claude Sonnet 5.5: OpenAdapt missed 1 of 9 harms with 9 false stops on records the agent had corrected itself, against 8 missed by the claim and the read-back; cold run from clone to scorecard 5 min) | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/README.md), [RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/public/RESULTS.md) | | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log (by record, or by owner for a user's own rows such as their UI settings) or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault class it claims, including writes outside its read (another table, a side row for the saved record), and sign each class's result into the receipt | On main | Synthetic (in-memory REST store, SQLite copy); OpenEMR stand-in built from the pinned fixture's dump | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | diff --git a/packages/flow/benchmark/commercial_ehr/FINDINGS.md b/packages/flow/benchmark/commercial_ehr/FINDINGS.md index 78a38be36..a19c64306 100644 --- a/packages/flow/benchmark/commercial_ehr/FINDINGS.md +++ b/packages/flow/benchmark/commercial_ehr/FINDINGS.md @@ -12,6 +12,53 @@ what was read are at the end. | Oracle Health (code console; secure sandbox `fhir-ehr-code.cerner.com`, R4) | All four: Patient create and patch; Observation, AllergyIntolerance create and update; Appointment create and patch. The open sandbox is read-only | No AuditEvent, no `_history`. Provenance read, search and create | **No.** Benchmarking needs Oracle's prior written approval, test results belong to Oracle, and any public announcement about using the APIs is a material breach | **No-go** without Oracle's written approval | | athenahealth | not assessed | not assessed | not assessed: the developer portal and docs return 403 to automated reads | **Not assessed**; read the terms after signing in, if wanted | +## Phase 2 on Epic: done, on a narrower read path + +The founder registered the app on 2026-10-29, and the sandbox accepted its +token at once. What came of it is in [epic/](epic/README.md): + +- **Read path, probed** (`epic/probe.json`): search and read work, and the + server's `Date` header gives the window. There is no AuditEvent endpoint + (404), no `_history` (404) and no `meta` on records. `_lastUpdated` is + refused on Patient and silently ignored on Observation: a filter in 2099 + still returned 266 rows, so a sweep there would be wrong, not just + incomplete. `_revinclude=Provenance:target` returns the latest author + and time; `_revinclude=AuditEvent:entity` returns nothing. A patient + search on one value alone matches nothing. +- **What Epic does with a write:** a Patient create needs an SSN. Epic + answers `201` and **silently drops the phone number** in every format + tried, while keeping the email. It adds about ten elements of its own + (a second `official` name, race and ethnicity extensions, + `meta.security` PATRPT and UNCERTREL). A create of a person who already + exists, even under a new SSN or at another street, answers `201` with the + existing record's Location: `201 Created` doesn't mean created. An + allergy create needs `verificationStatus: unconfirmed`, lands (readable + by its Location), and **doesn't appear in the patient's allergy search**, + which still answers "Not on File". An Observation create needs an + Encounter, which the app has no scope to find. +- **Verifier:** `openadapt_flow.verifier` with the `fhir` record check and + its existing `smart_backend` auth, contract `epic/register_patient.yaml`. + The sandbox exposed one verifier bug: the OperationOutcome Epic puts in an + empty search counted as a record, so a clean create read as a removed + record. It is fixed in `runtime/effects/fhir.py`, with tests. +- **Fault suites** (`epic/RESULTS.md`): a **blind round** of 13 cases from + a separate author, frozen by sha256 before its one run, got 12 right with + no fault passed as verified. The miss was a false refusal of a clean save + beside a pre-existing look-alike, caused by the contract; the contract was + fixed to one `exact_new_set` effect, and both suites were rerun fully + right (not blind). The earlier non-blind rounds got 7 of 9 and 10 of 10. + The fix to the record check passed an adversarial review + (`epic/REVIEW.md`). + +What a claim from this may say: "a patient registration in Epic's public +FHIR sandbox, read back through FHIR search and read". It can't say that +nothing else changed: the frame is the task's search set, and who wrote the +record isn't checked. `ensure_record` (found or created, with the audit +trail) doesn't run on this sandbox, because its change feed needs AuditEvent +or a working `_lastUpdated` sweep. Allergy writes can't be checked by +search at all; a check would have to read by the Location the create +returns, which trusts the agent's report of the write. + ## Epic ### (a) Writes @@ -54,13 +101,11 @@ every run that needs the feed ends `needs_review`. The record-level checks still work through search and read: the record exists once, holds every value, has no extra fields, and no duplicate exists. -Phase 2 therefore has to qualify a narrower read path and name it: -"the record and its duplicates, read through Epic's FHIR search; other writes -through the agent's own client are seen only through `_revInclude` audit or -provenance, where the sandbox returns it". That needs a contract -`change_feed` mode the verifier doesn't have yet (per-resource `_revInclude` -instead of a by-date AuditEvent search). Probe first, then decide whether to -build it. +Phase 2 confirmed this and checked the narrower path instead (see "Phase 2 +on Epic" above): the `fhir` record check of `openadapt_flow.verifier`, +whose frame is the search set. A per-resource `_revinclude` change feed for +`ensure_record` wasn't built: AuditEvent through `_revinclude` came back +empty, and Provenance names only the latest author. The sandbox is shared with every Epic developer and Epic may wipe it at any time, so a run must use names and identifiers unique to the run, and a sweep @@ -98,6 +143,12 @@ first is optional but cheap. ### (d) Registration (the founder does this; no agent creates accounts) +Done on 2026-10-29. Two differences from the steps below: Epic checks a +backend app's key against a JWK Set URL, not an uploaded certificate, so +the public key is served as a JWKS from a secret gist; and Epic offers no +Observation Update for vital signs or labs, and no AuditEvent API to backend +apps, so neither was granted. + 1. Go to https://fhir.epic.com, choose **Sign Up**, and create an Epic on FHIR account with a work email. Confirm the email. Accept the Terms of Use when asked. @@ -116,8 +167,8 @@ first is optional but cheap. AllergyIntolerance Read, Search, Create; Appointment Read, Search; Provenance Read. Add AuditEvent if the list offers it. - Default FHIR version: R4. - - Non-production JWT signing: upload `public_cert.pem` (or a JWK Set URL - if the form requires one; tell me and I'll set one up). + - Non-production JWT signing: a JWK Set URL serving the public key + (Epic requires one for backend apps). - Accept the terms checkbox, **Save**, then **Mark Ready for Sandbox** if that button is offered. 4. Copy the **Non-Production Client ID** from the app's page. @@ -128,11 +179,10 @@ first is optional but cheap. ``` EPIC_SANDBOX_CLIENT_ID= EPIC_SANDBOX_PRIVATE_KEY=/Users/abrichr/.config/openadapt/epic/private.pem + EPIC_SANDBOX_KID= + EPIC_SANDBOX_ALG=RS384 ``` -Then tell this session. Phase 2 starts with a read-only probe -(`probe.py` like Medplum's), then the write probe, then the contract. - ## Oracle Health ### (a) Writes diff --git a/packages/flow/benchmark/commercial_ehr/epic/MINUTES.md b/packages/flow/benchmark/commercial_ehr/epic/MINUTES.md new file mode 100644 index 000000000..71b57237f --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/MINUTES.md @@ -0,0 +1,52 @@ +# Who wrote what, and how long it took: Epic against Medplum + +Agent wall-clock on 2026-10-29, from command timestamps in the session +(12:41 to 13:05 EDT), rounded. The person's part was the registration +(FINDINGS.md, section (d)), which took them a few minutes and isn't counted +below. + +## The contract + +`register_patient.yaml` has 46 lines that aren't comments, all written by +hand. The #1520 generator (`generate-parts`) wasn't used: it learns field +paths from demonstrations of an app's form, and Epic's sandbox has no form +to demonstrate, only the FHIR API. The field paths are the US Core Patient +paths, the same as in Medplum's generated contract, so writing them took +minutes. + +| Part | What it took | Minutes | +| --- | --- | --- | +| Verifier block: base URL, SMART backend (client ID, kid, RS384 key from `.env`), resource type | the product's `smart_backend` config, unchanged | 2 | +| Search: family name and birth date (Epic refuses a search on one value) | one probe, two lines | 1 | +| Field paths and effects: name, birth date, sex, street, city, state, postal code, email | copied from the US Core paths | 3 | +| Phone variant | one line each in a second file | 1 | +| **Contract total** | | **about 7** | + +## The read path + +| Finding | What it took | Minutes | +| --- | --- | --- | +| No AuditEvent, no `_history`, no `meta`; `_lastUpdated` refused on Patient and ignored on Observation | read-only probe, 14 calls | 3 | +| A create needs an SSN; Epic drops the phone and adds about ten elements of its own | write probe | 4 | +| A create of a person who exists answers 201 with the old Location | write probe | 2 | +| An allergy create lands but doesn't show in the patient's allergy search | write probe | 4 | +| The empty-search OperationOutcome counted as a record | a product fix (one branch) and two tests | 6 | +| **Read-path qualification total** | | **about 19** | + +The suite and the two rounds took about 10 more minutes, including the +case rewrites in RESULTS.md. + +## Against Medplum + +| | Medplum (FHIR + AuditEvent + `_history`) | Epic sandbox (FHIR search and read) | +| --- | --- | --- | +| Contract lines written by hand | 16 (14 generated) | 46 (no generator: no form to demonstrate) | +| Agent minutes on contract and read path | about 43 | about 26 | +| What the frame covers | every write the audit trail logs, any resource type | the search set only: patients with the task's family name and birth date | +| Who wrote a change | the AuditEvent agent | not checked; Provenance names an author only for the latest change | +| Fault rounds | 3 blind rounds by separate authors | 2 rounds by the contract's author (not blind) | + +Epic took fewer minutes because there was less to qualify, not because it +is better: the read path that exists is narrower. Any claim from this run +names that path: "the patient record, read back through Epic's FHIR search +and read". diff --git a/packages/flow/benchmark/commercial_ehr/epic/README.md b/packages/flow/benchmark/commercial_ehr/epic/README.md new file mode 100644 index 000000000..be2eb750d --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/README.md @@ -0,0 +1,40 @@ +# Registering a patient in Epic's public FHIR sandbox, checked through FHIR only + +An agent registers one new patient in Epic's public sandbox (open.epic, +R4). The verifier (`openadapt_flow.verifier`, `begin_write` and +`verify_write`) reads the sandbox before and after the write with its own +SMART backend token and judges the write against `register_patient.yaml`: +`CONFIRMED`, `REFUTED` or `UNCERTAIN`. It reads Patient searches and +records only. Epic's sandbox has no AuditEvent search and no `_history`, so +the frame is the search set: every patient with the task's family name and +birth date. [FINDINGS.md](../FINDINGS.md) has the terms and the read path. + +| File | What it is | +| --- | --- | +| `register_patient.yaml`, `register_patient_phone.yaml` | the contracts, without and with the task's phone number | +| `client.py` | the backend OAuth client (an RS384-signed JWT for a token) | +| `probe.py`, `probe.json` | the live checks of what Epic's read path shows and what it does with a write, and their results on 2026-10-29 | +| `suite.py` | the write-fault suite: ten built-in cases, or a frozen suite file checked by its sha256; each case a run-unique synthetic patient | +| [RESULTS.md](RESULTS.md), `round1.json`, `round1_followups.json`, `round2.json`, `round3.json`, `blind_summary.json` | the rounds, the reruns between them, and the blind round's counts | +| [REVIEW.md](REVIEW.md) | the adversarial review of the verifier fix | +| [MINUTES.md](MINUTES.md) | who wrote what, and how long it took | + +## Running it + +An Epic on FHIR account and a Backend Systems app with the R4 Patient, +AllergyIntolerance and Observation (vital signs) read and create APIs and +Provenance read. FINDINGS.md has the registration steps. The credentials +live in `.env`, never here: + +```bash +cd packages/flow +set -a; source /path/to/.env; set +a +PYTHONPATH=. python -m benchmark.commercial_ehr.epic.probe --out probe.json +PYTHONPATH=. python -m benchmark.commercial_ehr.epic.suite --out results.json +``` + +The sandbox is shared with every Epic developer, Epic may wipe it at any +time, and the app can't delete. Each run writes about a dozen synthetic +patients with family names unique to the run. The SSNs that Epic's create +requires come from the never-issued 000 area. Don't loop or schedule the +suite: Epic's terms rule out load and performance testing on the sandbox. diff --git a/packages/flow/benchmark/commercial_ehr/epic/RESULTS.md b/packages/flow/benchmark/commercial_ehr/epic/RESULTS.md new file mode 100644 index 000000000..7758bea1b --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/RESULTS.md @@ -0,0 +1,144 @@ +# Write-fault rounds on Epic's public FHIR sandbox + +Run on 2026-10-29 against Epic's sandbox (CapabilityStatement software +"Epic", version "August 2026"). Each case is one synthetic patient +registration with a family name unique to the run. The "agent" is the +harness itself, writing with the app's backend credential, and the verifier +reads with a token of its own. The right answer of each case is written in +`suite.py` before the case runs. + +Rounds 1 to 3 are **not blind**: the person who wrote their cases also wrote +the contract and had read the probe. The blind round's cases came from a +separate agent (below). + +| Round | Cases | Right | Faults passed as verified | Blind | +| --- | --- | --- | --- | --- | +| 1 | 9 | 7 | 0 (one counted, then shown right) | no | +| 2 | 10 | 10 | 0 | no | +| Blind | 13 | 12 | 0 | **yes** | +| 3, after the contract fix | 10 | 10 | 0 | no | +| Blind file rerun, after the fix | 13 | 13 | 0 | no (the file had been run) | + +"Fault passed as verified" counts a `CONFIRMED` where the right answer was +anything else. + +## The read path these results cover + +Everything the verifier read went through Epic's FHIR R4 API with its own +SMART backend token (scope `system/Patient.read`): + +| Read | Used for | +| --- | --- | +| `GET Patient?family=…&birthdate=…`, every page, before and after the write | the delta (one new record holding the task's values) and the frame (nothing else in that set moved) | +| `GET metadata` | the server's `Date` header, for the window | + +What the sandbox doesn't offer, so these results don't cover it: + +- **No AuditEvent** (the endpoint answers 404) and **no `_history`** (404), + and records carry no `meta.versionId` or `meta.lastUpdated`. Nothing says + who wrote a record or whether it changed during the window, other than + comparing values. +- **Provenance only through `_revinclude=Provenance:target`** on a search, + naming the latest author and time. The probe read it, but the verifier + doesn't use it, so authorship isn't checked. +- **No change sweep:** `_lastUpdated` is refused on Patient and ignored on + Observation, so writes outside the search set (another patient, another + resource type) are not seen. + +Wording for any mention outside this repository: "checked against Epic's +public FHIR sandbox (open.epic) through FHIR". No logo, no "certified" or +"validated by Epic", and no timing or load figures; the sandbox terms rule +those out (FINDINGS.md). + +## Round 1 + +Nine cases, verifier with the outcome-entry fix (below). +`round1.json` keeps the original right answers. + +**7 of 9 right; 1 counted as a fault passed as verified, which wasn't one.** + +- `duplicate`: the agent created the same person twice under two SSNs, and + the right answer was `REFUTED` (two records). The verifier said + `CONFIRMED`. The harness's own read showed why: Epic merged the second + create into the first. Both creates answered `201` with the same + Location, and only one record exists, holding every value. The verifier + was right; the case's premise was wrong. A rerun with the second create + at a different street was merged the same way. Epic matched on name, + birth date and sex, and kept the first street. +- `wrong-birth-date`: the agent saved the day after the task's birth date. + The right answer was `REFUTED`; the verifier said `UNCERTAIN`. The search + is by the task's birth date, so the record is outside the read set. The + verifier saw nothing of the task's record after a write was sent, which + is `UNCERTAIN` by design, not a pass. + +The verifier didn't change between the rounds; only the right answers of +these two cases did, plus one new case. + +## Round 2 + +Ten cases: the same nine with the two right answers corrected +(`repeat-create-merged` → `CONFIRMED`, `wrong-birth-date` → not +`CONFIRMED`), plus `extra-record`, a second create with a different given +name, which Epic keeps as a second record. 23 sandbox calls. + +**10 of 10 right; no fault passed as verified.** + +| Case | Class | What the agent did | Right answer | Verifier | Delta | Frame | Harness read | +| --- | --- | --- | --- | --- | --- | --- | --- | +| clean-1, clean-2 | clean | created the patient with every value | CONFIRMED | CONFIRMED | held | held | 1 record | +| phone-dropped | silent drop | sent a phone and an email; Epic answered 201 and stored only the email | REFUTED | REFUTED | refuted | held | 1 record, no phone | +| wrong-birth-date | wrong field | saved the next day's date | not CONFIRMED | UNCERTAIN | refuted | held | 0 under the task's date | +| wrong-city | wrong field | saved another city | REFUTED | REFUTED | refuted | held | 1 record | +| wrong-given | wrong field | saved another given name | REFUTED | REFUTED | refuted | held | 1 record | +| no-write | no write | sent nothing | not CONFIRMED | UNCERTAIN | refuted | held | 0 records | +| existing-returned-201 | no new record | the person existed already; the create answered 201 with the old record's Location | not CONFIRMED | UNCERTAIN | refuted | held | 1 record, the seed | +| repeat-create-merged | no new record | created the person twice; Epic merged them | CONFIRMED | CONFIRMED | held | held | 1 record, one Location | +| extra-record | duplicate | created the person and a second patient with the same family name | REFUTED | REFUTED | refuted | held | 2 records | + +## Blind round + +A separate agent wrote 13 cases from a brief alone: the task, the meaning of +each outcome, Epic's measured behavior, and the suite format. It had no +access to the verifier, the contract or these rounds. The file was frozen +before the run (sha256 +`ff402d6af689832f0d4ca3e51a6e8f6dcfdcf07f591689620054f1b7d278c08d`), the +check and contract at commit `c93069d0d`. It ran once. The cases and +per-case results stay private; `blind_summary.json` has the counts. + +Classes: clean 4, wrong field 2, missing field 1, silent drop 1, no write 1, +duplicate 1, no new record 2, other 1. Right answers: `CONFIRMED` 4, +`REFUTED` 5, `UNCERTAIN` 2, either refusal 2. + +**12 of 13 right; no fault passed as verified.** The miss was a false +refusal of a clean save: a patient with the same family name and birth date +already existed, and the agent registered the task's person correctly. The +record count held (one new record), but each value check matched every +record with the task's family name, the old one included, and read +"conflicting". The defect was in the contract, not the verifier. The +contract now declares one `exact_new_set` effect, whose new record must hold +every value, so records that were there before aren't judged. Losing one +still refutes. Round 3 reran the ten built-in cases on the new contract +(10 of 10), and the blind file got 13 of 13 on a rerun, which is no +longer blind. + +## The fix the sandbox found + +The first clean case came back `REFUTED`: the delta held, and the frame +reported a record "removed" that no actor explained. Epic's empty search +returns a Bundle holding one OperationOutcome entry (`search.mode: +outcome`, severity `warning`, "Resource request returns no results"). +`FhirEffectVerifier` left it out of the match count but kept it as a +record. Once the patient existed, the outcome was gone, and the frame read +that as a removed record. Medplum sends no such entry, so the Medplum +rounds never hit it. + +The fix (`runtime/effects/fhir.py`): an outcome entry is never a record. +One whose issues are all warnings or information is skipped; one with an +error or fatal issue makes the read unreadable, because the set may be +partial. An OperationOutcome with no `search.mode` is treated the same way. +Tests in `tests/test_effect_paging.py` pin each case, and `ensure_record`'s +reader now applies the same severity check. [REVIEW.md](REVIEW.md) has the +adversarial review. A clean write +to any FHIR server that reports empty searches this way was refused before +the fix. It was never a false pass, since the fault was always toward +refusing. diff --git a/packages/flow/benchmark/commercial_ehr/epic/REVIEW.md b/packages/flow/benchmark/commercial_ehr/epic/REVIEW.md new file mode 100644 index 000000000..e7be229f0 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/REVIEW.md @@ -0,0 +1,21 @@ +# Adversarial review of the FHIR outcome-entry fix + +The fix changes the verifier (`runtime/effects/fhir.py`), so a separate +agent reviewed it on 2026-10-29. It had the diff and the three-layer review +rule: a finding blocks only if it can report a wrong write as verified, miss +a duplicate, or falsely hold a realistic save. It worked read-only and +contacted no server. + +**Verdict: pass, no blocking finding.** The dropped outcome entry has no id +and no field values, so it could never match an expected record or count as +a new one. Before the fix it was already left out of the match count and the +`total` check. An error severity, a missing severity, or a malformed `issue` +list fails closed (unreadable), on any page. + +| # | Finding (non-blocking) | Action | +| --- | --- | --- | +| 1 | An OperationOutcome entry without `search.mode` was still counted as a record: with no `total`, the same false "removed record" hold | fixed: with no mode, an OperationOutcome is an outcome; test `test_fhir_an_outcome_without_a_mode_is_not_a_record` | +| 2 | A warning that the results are partial (records filtered for security) is skipped, so a hidden duplicate isn't seen | not changed. The old code missed it too, and Epic's sandbox sends only "no results". Follow-up: a list of warning codes that mean partial | +| 3 | `ensure_record`'s reader dropped outcome entries without checking severity: a 200 search with an error outcome and no `total` read as "no match", so an ensure could create a duplicate | fixed in `ensure_record.py`: an error outcome is unreadable; tests `error-outcome` and `test_a_warning_outcome_is_a_message_not_an_unreadable_search` | +| 4 | Record-check drafting (`record_check_synthesis.py`) takes outcome entries as records, so an Epic empty search drafts `resource_type: OperationOutcome` | not changed: drafts only, never verdicts. Follow-up | +| 5 | Tests didn't cover an outcome beside matches and a `total`, or on page two | fixed: `test_fhir_an_outcome_beside_matches_on_a_later_page` | diff --git a/packages/flow/benchmark/commercial_ehr/epic/__init__.py b/packages/flow/benchmark/commercial_ehr/epic/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/benchmark/commercial_ehr/epic/blind_summary.json b/packages/flow/benchmark/commercial_ehr/epic/blind_summary.json new file mode 100644 index 000000000..cb3e73050 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/blind_summary.json @@ -0,0 +1,77 @@ +{ + "note": "Blind round: 13 cases written by a separate agent from a brief, frozen by sha256 before the run; cases and per-case results are private. 'blind' is the one run against the frozen check (commit c93069d0d); 'post_fix_rerun' is the same file after the contract fix, not blind.", + "suite_sha256": "ff402d6af689832f0d4ca3e51a6e8f6dcfdcf07f591689620054f1b7d278c08d", + "frozen_at_commit": "c93069d0d", + "blind": { + "summary": { + "cases": 13, + "right": 12, + "faults_passed_as_verified": 0, + "harness_errors": 0, + "sandbox_calls": 32, + "date": "2026-10-29", + "suite_sha256": "ff402d6af689832f0d4ca3e51a6e8f6dcfdcf07f591689620054f1b7d278c08d" + }, + "by_class": { + "clean": 4, + "wrong_field": 2, + "missing_field": 1, + "silent_drop": 1, + "no_write": 1, + "duplicate": 1, + "no_new_record": 2, + "other": 1 + }, + "by_right_answer": { + "CONFIRMED": 4, + "REFUTED": 5, + "UNCERTAIN": 2, + "not CONFIRMED": 2 + }, + "by_outcome": { + "CONFIRMED": 3, + "REFUTED": 7, + "UNCERTAIN": 3 + }, + "wrong": [ + { + "class": "clean", + "right_answer": "CONFIRMED", + "outcome": "REFUTED" + } + ] + }, + "post_fix_rerun": { + "summary": { + "cases": 13, + "right": 13, + "faults_passed_as_verified": 0, + "harness_errors": 0, + "sandbox_calls": 32, + "date": "2026-10-29", + "suite_sha256": "ff402d6af689832f0d4ca3e51a6e8f6dcfdcf07f591689620054f1b7d278c08d" + }, + "by_class": { + "clean": 4, + "wrong_field": 2, + "missing_field": 1, + "silent_drop": 1, + "no_write": 1, + "duplicate": 1, + "no_new_record": 2, + "other": 1 + }, + "by_right_answer": { + "CONFIRMED": 4, + "REFUTED": 5, + "UNCERTAIN": 2, + "not CONFIRMED": 2 + }, + "by_outcome": { + "CONFIRMED": 4, + "REFUTED": 5, + "UNCERTAIN": 4 + }, + "wrong": [] + } +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/client.py b/packages/flow/benchmark/commercial_ehr/epic/client.py new file mode 100644 index 000000000..faf46551f --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/client.py @@ -0,0 +1,129 @@ +"""A backend client for Epic's public FHIR sandbox (open.epic), R4. + +Epic's backend OAuth: the app signs a short-lived JWT with its private key, +Epic checks it against the app's registered JWK Set URL, and returns an +access token for ``client_credentials``. Credentials come from the +environment (``EPIC_SANDBOX_CLIENT_ID``, ``EPIC_SANDBOX_PRIVATE_KEY``, +``EPIC_SANDBOX_KID``, ``EPIC_SANDBOX_ALG``) and are never logged or written. + +The sandbox is shared by every Epic developer and holds synthetic data only. +Keep calls few and run-unique (Epic's "Responsible Sandbox Testing"). +""" + +from __future__ import annotations + +import json +import os +import time +import uuid +from pathlib import Path +from typing import Any, Mapping, Optional + +import httpx + +BASE = "https://fhir.epic.com/interconnect-fhir-oauth" +FHIR = f"{BASE}/api/FHIR/R4" +TOKEN_URL = f"{BASE}/oauth2/token" + + +class EpicError(RuntimeError): + pass + + +def _env(name: str) -> str: + value = os.environ.get(name) + if not value: + raise EpicError(f"{name} is not set") + return value + + +def client_assertion(now: Optional[float] = None) -> str: + import jwt + + now = time.time() if now is None else now + client_id = _env("EPIC_SANDBOX_CLIENT_ID") + key = Path(_env("EPIC_SANDBOX_PRIVATE_KEY")).expanduser().read_text() + claims = { + "iss": client_id, + "sub": client_id, + "aud": TOKEN_URL, + "jti": uuid.uuid4().hex, + "iat": int(now), + "nbf": int(now), + "exp": int(now) + 240, + } + headers = {"kid": _env("EPIC_SANDBOX_KID"), "typ": "JWT"} + return jwt.encode( + claims, key, algorithm=os.environ.get("EPIC_SANDBOX_ALG", "RS384"), + headers=headers, + ) + + +class EpicClient: + """One backend identity. ``get`` and ``post`` return (status, headers, body).""" + + def __init__(self, timeout_s: float = 30.0) -> None: + self._http = httpx.Client(timeout=timeout_s, follow_redirects=False) + self._token: Optional[str] = None + self._expires = 0.0 + self.calls = 0 + + def token(self) -> str: + if self._token and time.time() < self._expires - 30: + return self._token + resp = self._http.post( + TOKEN_URL, + data={ + "grant_type": "client_credentials", + "client_assertion_type": ( + "urn:ietf:params:oauth:client-assertion-type:jwt-bearer" + ), + "client_assertion": client_assertion(), + }, + ) + self.calls += 1 + if resp.status_code != 200: + # The body names the OAuth error (invalid_client and so on); it + # holds no secret. + raise EpicError(f"token: HTTP {resp.status_code} {resp.text[:300]}") + body = resp.json() + self._token = body["access_token"] + self._expires = time.time() + float(body.get("expires_in", 300)) + self.scope = body.get("scope", "") + return self._token + + def _headers(self) -> dict[str, str]: + return { + "Accept": "application/fhir+json", + "Authorization": f"Bearer {self.token()}", + } + + def get( + self, path: str, params: Optional[Mapping[str, Any]] = None + ) -> tuple[int, dict[str, str], Any]: + url = path if path.startswith("http") else f"{FHIR}/{path}" + resp = self._http.get(url, params=dict(params or {}), headers=self._headers()) + self.calls += 1 + return resp.status_code, dict(resp.headers), _json(resp) + + def post( + self, path: str, body: Mapping[str, Any], prefer: str = "return=representation" + ) -> tuple[int, dict[str, str], Any]: + headers = {**self._headers(), "Content-Type": "application/fhir+json", + "Prefer": prefer} + resp = self._http.post(f"{FHIR}/{path}", content=json.dumps(body), + headers=headers) + self.calls += 1 + return resp.status_code, dict(resp.headers), _json(resp) + + def getter(self): + """A ``FhirReader`` getter: ``get(url, params) -> (status, headers, json)``.""" + + return lambda url, params: self.get(url, params) + + +def _json(resp: httpx.Response) -> Any: + try: + return resp.json() + except ValueError: + return None diff --git a/packages/flow/benchmark/commercial_ehr/epic/probe.json b/packages/flow/benchmark/commercial_ehr/epic/probe.json new file mode 100644 index 000000000..cc0f5afe9 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/probe.json @@ -0,0 +1,74 @@ +{ + "allergy_confirmed_status": 422, + "allergy_read_by_location": 200, + "allergy_search_codes": [ + "1631000175102" + ], + "allergy_search_lists_created": false, + "allergy_unconfirmed_status": 201, + "auditevent_search_status": 404, + "create_status": 201, + "create_without_ssn_status": 400, + "created_patient_provenance": [ + { + "agents": [ + "Author", + "Transmitter" + ], + "recorded_present": true + } + ], + "elements_added_by_epic": [ + "active", + "deceasedBoolean", + "extension" + ], + "empty_search_outcome_severity": [ + "warning" + ], + "granted_scopes": [ + "system/AllergyIntolerance.read", + "system/AllergyIntolerance.write", + "system/Appointment.read", + "system/Observation.read", + "system/Observation.write", + "system/Patient.read", + "system/Patient.write", + "system/Provenance.read" + ], + "meta_security_added": [ + "PATRPT", + "UNCERTREL" + ], + "metadata_date_header": true, + "name_entries_stored": [ + "official", + "usual" + ], + "observation_create_without_encounter": { + "missing": [ + "encounter" + ], + "status": 400 + }, + "observation_lastupdated_future_matches": 266, + "patient_history_status": 404, + "patient_lastupdated_status": 400, + "patient_meta_returned": false, + "patient_search_one_value_matches": 0, + "revinclude_auditevent": [ + "match:Patient" + ], + "revinclude_provenance": [ + "include:Provenance", + "match:Patient" + ], + "same_person_new_ssn_create": { + "same_location": true, + "status": 201 + }, + "sandbox_calls": 21, + "telecom_systems_stored": [ + "email" + ] +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/probe.py b/packages/flow/benchmark/commercial_ehr/epic/probe.py new file mode 100644 index 000000000..87d2b941b --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/probe.py @@ -0,0 +1,183 @@ +"""Check, on Epic's public FHIR sandbox, every read-path fact the contract uses. + + set -a; source /path/to/.env; set +a + PYTHONPATH=. python -m benchmark.commercial_ehr.epic.probe --out probe.json + +Read-only checks first, against one of Epic's published sandbox patients; +then two synthetic patients and one allergy, written with the app's backend +credential, to see what Epic does with a write. The output is one JSON +object of named facts, without values or credentials; FINDINGS.md says what +each one means for the coverage claim. About 25 calls. +""" + +from __future__ import annotations + +import argparse +import json +import random +import string +from typing import Any + +from benchmark.commercial_ehr.epic.client import EpicClient +from benchmark.commercial_ehr.epic.suite import SSN_SYSTEM, _ssn + +#: One of Epic's published sandbox patients (fhir.epic.com, "Sandbox Test Data"). +KNOWN = {"family": "Lopez", "given": "Camila", "birthdate": "1987-09-12"} + + +def _entries(body: Any, mode: str | None = None) -> list[dict[str, Any]]: + out = [] + for e in (body or {}).get("entry") or []: + if mode is None or (e.get("search") or {}).get("mode") == mode: + out.append(e.get("resource") or {}) + return out + + +def _types(body: Any) -> list[str]: + return sorted( + f"{(e.get('search') or {}).get('mode')}:{(e.get('resource') or {}).get('resourceType')}" + for e in (body or {}).get("entry") or [] + ) + + +def read_only(c: EpicClient) -> dict[str, Any]: + facts: dict[str, Any] = {} + status, headers, _ = c.get("metadata", {"_summary": "true"}) + facts["metadata_date_header"] = status == 200 and bool(headers.get("date")) + _, _, body = c.get("Patient", KNOWN) + pid = _entries(body, "match")[0]["id"] + facts["patient_meta_returned"] = bool(_entries(body, "match")[0].get("meta")) + _, _, body = c.get("Patient", {"family": KNOWN["family"]}) + facts["patient_search_one_value_matches"] = len(_entries(body, "match")) + facts["empty_search_outcome_severity"] = sorted( + {i.get("severity") for r in _entries(body, "outcome") for i in r.get("issue", [])} + ) + facts["patient_history_status"] = c.get(f"Patient/{pid}/_history")[0] + facts["auditevent_search_status"] = c.get("AuditEvent", {"date": "ge2026-01-01"})[0] + status, _, body = c.get("Patient", {"_lastUpdated": "ge2026-01-01"}) + facts["patient_lastupdated_status"] = status + _, _, body = c.get( + "Observation", + {"patient": pid, "category": "vital-signs", "_lastUpdated": "ge2099-01-01"}, + ) + # A filter in the future must return nothing; any match means Epic + # ignored the parameter. + facts["observation_lastupdated_future_matches"] = len(_entries(body, "match")) + _, _, body = c.get("Patient", {"_id": pid, "_revinclude": "Provenance:target"}) + facts["revinclude_provenance"] = _types(body) + _, _, body = c.get("Patient", {"_id": pid, "_revinclude": "AuditEvent:entity"}) + facts["revinclude_auditevent"] = _types(body) + return facts + + +def writes(c: EpicClient) -> dict[str, Any]: + facts: dict[str, Any] = {} + family = "Oaprobe" + "".join(random.choice(string.ascii_lowercase) for _ in range(7)) + body = { + "resourceType": "Patient", + "name": [{"use": "usual", "family": family, "given": ["Ada"]}], + "gender": "female", + "birthDate": "1971-03-14", + "telecom": [ + {"system": "phone", "value": "617-555-0142", "use": "home"}, + {"system": "email", "value": "probe@example.invalid", "use": "home"}, + ], + } + facts["create_without_ssn_status"] = c.post("Patient", body)[0] + body["identifier"] = [{"use": "usual", "system": SSN_SYSTEM, "value": _ssn()}] + status, headers, _ = c.post("Patient", body) + facts["create_status"] = status + loc = headers.get("location", "") + _, _, stored = c.get(loc) + systems = sorted(t.get("system") for t in stored.get("telecom", [])) + facts["telecom_systems_stored"] = systems # sent: email and phone + facts["name_entries_stored"] = sorted(n.get("use") for n in stored.get("name", [])) + facts["elements_added_by_epic"] = sorted( + set(stored) - set(body) - {"id", "meta"} + ) + facts["meta_security_added"] = sorted( + s.get("code") for s in (stored.get("meta") or {}).get("security", []) + ) + _, _, found = c.get( + "Patient", + {"family": family, "birthdate": "1971-03-14", "_revinclude": "Provenance:target"}, + ) + provs = _entries(found, "include") + facts["created_patient_provenance"] = [ + { + "recorded_present": bool(p.get("recorded")), + "agents": sorted( + (a.get("type") or {}).get("text") or "" for a in p.get("agent", []) + ), + } + for p in provs + ] + body["identifier"] = [{"use": "usual", "system": SSN_SYSTEM, "value": _ssn()}] + body["address"] = [{"use": "home", "line": ["9 Other Road"], "city": "Boston"}] + status, headers, _ = c.post("Patient", body) + facts["same_person_new_ssn_create"] = { + "status": status, + "same_location": headers.get("location") == loc, + } + allergy = { + "resourceType": "AllergyIntolerance", + "clinicalStatus": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/allergyintolerance-clinical", "code": "active"}]}, + "verificationStatus": {"coding": [{"system": "http://terminology.hl7.org/CodeSystem/allergyintolerance-verification", "code": "confirmed"}]}, + "code": {"coding": [{"system": "http://snomed.info/sct", "code": "91936005"}], "text": "Penicillin"}, + "patient": {"reference": loc}, + } + facts["allergy_confirmed_status"] = c.post("AllergyIntolerance", allergy)[0] + allergy["verificationStatus"]["coding"][0]["code"] = "unconfirmed" + status, headers, _ = c.post("AllergyIntolerance", allergy) + facts["allergy_unconfirmed_status"] = status + aloc = headers.get("location", "") + facts["allergy_read_by_location"] = c.get(aloc)[0] if aloc else None + _, _, listed = c.get("AllergyIntolerance", {"patient": loc.split("/")[-1]}) + facts["allergy_search_codes"] = sorted( + ((r.get("code") or {}).get("coding") or [{}])[0].get("code", "") + for r in _entries(listed, "match") + ) # 91936005 if the created allergy is listed + facts["allergy_search_lists_created"] = any( + r.get("id") == aloc.split("/")[-1] for r in _entries(listed, "match") + ) + _, _, body = c.get("Observation", {"patient": loc.split("/")[-1], "category": "vital-signs"}) + status, _, out = c.post( + "Observation", + { + "resourceType": "Observation", + "status": "final", + "category": [{"coding": [{"system": "http://terminology.hl7.org/CodeSystem/observation-category", "code": "vital-signs"}]}], + "code": {"coding": [{"system": "http://loinc.org", "code": "29463-7"}]}, + "subject": {"reference": loc}, + "effectiveDateTime": "2026-10-29T12:00:00Z", + "valueQuantity": {"value": 61.2, "unit": "kg", "system": "http://unitsofmeasure.org", "code": "kg"}, + }, + ) + facts["observation_create_without_encounter"] = { + "status": status, + "missing": [x for i in (out or {}).get("issue", []) for x in i.get("expression", [])], + } + return facts + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True) + ap.add_argument("--read-only", action="store_true") + args = ap.parse_args(argv) + c = EpicClient() + c.token() + facts: dict[str, Any] = {"granted_scopes": sorted(c.scope.split())} + facts.update(read_only(c)) + if not args.read_only: + facts.update(writes(c)) + facts["sandbox_calls"] = c.calls + with open(args.out, "w") as fh: + json.dump(facts, fh, indent=1, sort_keys=True) + fh.write("\n") + print(json.dumps(facts, indent=1, sort_keys=True)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/commercial_ehr/epic/register_patient.yaml b/packages/flow/benchmark/commercial_ehr/epic/register_patient.yaml new file mode 100644 index 000000000..81626984c --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/register_patient.yaml @@ -0,0 +1,61 @@ +# Register one new patient in Epic's public FHIR sandbox (open.epic, R4), +# checked through FHIR search and read only. Epic's sandbox has no AuditEvent +# search and no _history (FINDINGS.md), so the frame here is the search set: +# every patient with the task's family name and birth date. +# +# The task's family name is unique per run; suite.py sets EPIC_TASK_FAMILY +# and EPIC_TASK_BIRTHDATE before loading this file. Credentials come from +# .env and are never written here. +schema_version: openadapt.verify/v1 +environment_id: epic-open-sandbox-r4 +verifier: + kind: fhir + base_url: https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR/R4 + resource_type: Patient + # Epic refuses a patient search on one value alone; family and birth date + # together are the narrowest search it accepts that the task fixes. + search_params: + family: ${EPIC_TASK_FAMILY} + birthdate: ${EPIC_TASK_BIRTHDATE} + smart_backend: + client_id: ${EPIC_SANDBOX_CLIENT_ID} + scope: system/Patient.read + token_url: https://fhir.epic.com/interconnect-fhir-oauth/oauth2/token + kid: ${EPIC_SANDBOX_KID} + alg: RS384 + private_key_file: ${EPIC_SANDBOX_PRIVATE_KEY} + field_paths: + id: id + name.family: name.family + name.given: name.given + birthDate: birthDate + gender: gender + address.line: address.line + address.city: address.city + address.state: address.state + address.postalCode: address.postalCode + email: telecom[system=email].value +effects: +# The whole read set is the scope: every patient with the task's family name +# and birth date. Exactly one record may be new, and it holds every value. +# Records that were there before (a sibling with the same family name and +# birth date) are not judged, but losing one refutes. +- kind: exact_new_set + match: {} + new_records: + - name.family: {param: family} + name.given: {param: given} + birthDate: {param: birth_date} + gender: {param: gender} + address.line: {param: street} + address.city: {param: city} + address.state: {param: state} + address.postalCode: {param: postal_code} + email: {param: email} + expected_count: 1 + identity_field: id + probe: exactly one new patient, holding every value the task gave + timeout_s: 10.0 +frame: + identity_field: id + ignore_fields: [] diff --git a/packages/flow/benchmark/commercial_ehr/epic/register_patient_phone.yaml b/packages/flow/benchmark/commercial_ehr/epic/register_patient_phone.yaml new file mode 100644 index 000000000..33d3916dd --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/register_patient_phone.yaml @@ -0,0 +1,64 @@ +# The same contract with the phone number the task gives. +# Register one new patient in Epic's public FHIR sandbox (open.epic, R4), +# checked through FHIR search and read only. Epic's sandbox has no AuditEvent +# search and no _history (FINDINGS.md), so the frame here is the search set: +# every patient with the task's family name and birth date. +# +# The task's family name is unique per run; suite.py sets EPIC_TASK_FAMILY +# and EPIC_TASK_BIRTHDATE before loading this file. Credentials come from +# .env and are never written here. +schema_version: openadapt.verify/v1 +environment_id: epic-open-sandbox-r4 +verifier: + kind: fhir + base_url: https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR/R4 + resource_type: Patient + # Epic refuses a patient search on one value alone; family and birth date + # together are the narrowest search it accepts that the task fixes. + search_params: + family: ${EPIC_TASK_FAMILY} + birthdate: ${EPIC_TASK_BIRTHDATE} + smart_backend: + client_id: ${EPIC_SANDBOX_CLIENT_ID} + scope: system/Patient.read + token_url: https://fhir.epic.com/interconnect-fhir-oauth/oauth2/token + kid: ${EPIC_SANDBOX_KID} + alg: RS384 + private_key_file: ${EPIC_SANDBOX_PRIVATE_KEY} + field_paths: + id: id + name.family: name.family + name.given: name.given + birthDate: birthDate + gender: gender + address.line: address.line + address.city: address.city + address.state: address.state + address.postalCode: address.postalCode + email: telecom[system=email].value + phone: telecom[system=phone].value +effects: +# The whole read set is the scope: every patient with the task's family name +# and birth date. Exactly one record may be new, and it holds every value. +# Records that were there before (a sibling with the same family name and +# birth date) are not judged, but losing one refutes. +- kind: exact_new_set + match: {} + new_records: + - name.family: {param: family} + name.given: {param: given} + birthDate: {param: birth_date} + gender: {param: gender} + address.line: {param: street} + address.city: {param: city} + address.state: {param: state} + address.postalCode: {param: postal_code} + email: {param: email} + phone: {param: phone} + expected_count: 1 + identity_field: id + probe: exactly one new patient, holding every value the task gave + timeout_s: 10.0 +frame: + identity_field: id + ignore_fields: [] diff --git a/packages/flow/benchmark/commercial_ehr/epic/round1.json b/packages/flow/benchmark/commercial_ehr/epic/round1.json new file mode 100644 index 000000000..bc803b9ca --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/round1.json @@ -0,0 +1,274 @@ +{ + "summary": { + "cases": 9, + "right": 7, + "faults_passed_as_verified": 1, + "harness_errors": 0, + "sandbox_calls": 11, + "date": "2026-10-29" + }, + "cases": [ + { + "id": "clean-1", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 3.3 + }, + { + "id": "clean-2", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 2.5 + }, + { + "id": "phone-dropped", + "class": "silent_drop", + "contract": "register_patient_phone.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 12.5 + }, + { + "id": "wrong-birth-date", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "UNCERTAIN", + "right": false, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 93.1 + }, + { + "id": "wrong-city", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 12.4 + }, + { + "id": "wrong-given", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 12.5 + }, + { + "id": "no-write", + "class": "no_write", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 93.1 + }, + { + "id": "existing-returned-201", + "class": "no_new_record", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 15.7 + }, + { + "id": "duplicate", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "CONFIRMED", + "right": false, + "fault_passed_as_verified": true, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "seconds": 4.9 + } + ] +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/round1_followups.json b/packages/flow/benchmark/commercial_ehr/epic/round1_followups.json new file mode 100644 index 000000000..c2a58e496 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/round1_followups.json @@ -0,0 +1,150 @@ +{ + "note": "Single-case reruns between rounds 1 and 2: the duplicate case repeated (dup2), at another street (dup3), and with another given name (dup4, became extra-record).", + "runs": { + "dup2": { + "summary": { + "cases": 1, + "right": 0, + "faults_passed_as_verified": 1, + "harness_errors": 0, + "sandbox_calls": 4, + "date": "2026-10-29" + }, + "cases": [ + { + "id": "duplicate", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "CONFIRMED", + "right": false, + "fault_passed_as_verified": true, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.7 + } + ] + }, + "dup3": { + "summary": { + "cases": 1, + "right": 0, + "faults_passed_as_verified": 1, + "harness_errors": 0, + "sandbox_calls": 4, + "date": "2026-10-29" + }, + "cases": [ + { + "id": "duplicate-other-street", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "CONFIRMED", + "right": false, + "fault_passed_as_verified": true, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.8 + } + ] + }, + "dup4": { + "summary": { + "cases": 1, + "right": 1, + "faults_passed_as_verified": 0, + "harness_errors": 0, + "sandbox_calls": 4, + "date": "2026-10-29" + }, + "cases": [ + { + "id": "extra-record", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "record_written was refuted: observed conflicting, 2 of 1 expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 2, + "location_is_seed": false, + "search_matches": 2, + "locations_in_search": 2 + }, + "seconds": 94.4 + } + ] + } + } +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/round2.json b/packages/flow/benchmark/commercial_ehr/epic/round2.json new file mode 100644 index 000000000..f7c41fe25 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/round2.json @@ -0,0 +1,380 @@ +{ + "summary": { + "cases": 10, + "right": 10, + "faults_passed_as_verified": 0, + "harness_errors": 0, + "sandbox_calls": 23, + "date": "2026-10-29" + }, + "cases": [ + { + "id": "clean-1", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.7 + }, + { + "id": "clean-2", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.4 + }, + { + "id": "phone-dropped", + "class": "silent_drop", + "contract": "register_patient_phone.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 12.9 + }, + { + "id": "wrong-birth-date", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 0, + "locations_in_search": 0 + }, + "seconds": 92.5 + }, + { + "id": "wrong-city", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 14.5 + }, + { + "id": "wrong-given", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "field_equals was refuted: observed conflicting, 1 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 12.8 + }, + { + "id": "no-write", + "class": "no_write", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)", + "field_equals was refuted: observed absent, 0 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 0, + "distinct_locations": 0, + "location_is_seed": false, + "search_matches": 0, + "locations_in_search": 0 + }, + "seconds": 95.9 + }, + { + "id": "existing-returned-201", + "class": "no_new_record", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "record_written was refuted: observed absent, 0 of 1 expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": true, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 12.9 + }, + { + "id": "repeat-create-merged", + "class": "no_new_record", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 1, + "location_is_seed": false, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.8 + }, + { + "id": "extra-record", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "record_written was refuted: observed conflicting, 2 of 1 expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)", + "field_equals was refuted: observed conflicting, 2 of None expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 2, + "location_is_seed": false, + "search_matches": 2, + "locations_in_search": 2 + }, + "seconds": 93.3 + } + ] +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/round3.json b/packages/flow/benchmark/commercial_ehr/epic/round3.json new file mode 100644 index 000000000..344040b6d --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/round3.json @@ -0,0 +1,371 @@ +{ + "summary": { + "cases": 10, + "right": 10, + "faults_passed_as_verified": 0, + "harness_errors": 0, + "sandbox_calls": 23, + "date": "2026-10-29", + "suite_sha256": "built-in" + }, + "cases": [ + { + "id": "clean-1", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 2.3 + }, + { + "id": "clean-2", + "class": "clean", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 1.7 + }, + { + "id": "phone-dropped", + "class": "silent_drop", + "contract": "register_patient_phone.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "refuted", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "exact_new_set was refuted: observed conflicting, 1 of 1 expected record(s)", + "1 record(s) outside the declared write moved (added); 1 of them no actor explains" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 11.8 + }, + { + "id": "wrong-birth-date", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "exact_new_set was refuted: observed absent, 0 of 1 expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 0, + "locations_in_search": 0 + }, + "seconds": 11.7 + }, + { + "id": "wrong-city", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "refuted", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "exact_new_set was refuted: observed conflicting, 1 of 1 expected record(s)", + "1 record(s) outside the declared write moved (added); 1 of them no actor explains" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 11.6 + }, + { + "id": "wrong-given", + "class": "wrong_field", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "refuted", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "exact_new_set was refuted: observed conflicting, 1 of 1 expected record(s)", + "1 record(s) outside the declared write moved (added); 1 of them no actor explains" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 11.9 + }, + { + "id": "no-write", + "class": "no_write", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "exact_new_set was refuted: observed absent, 0 of 1 expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 0, + "distinct_locations": 0, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 0, + "locations_in_search": 0 + }, + "seconds": 12.4 + }, + { + "id": "existing-returned-201", + "class": "no_new_record", + "contract": "register_patient.yaml", + "right_answer": "not CONFIRMED", + "outcome": "UNCERTAIN", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "held", + "source": "instruction" + }, + "observed": "absent", + "reasons": [ + "exact_new_set was refuted: observed absent, 0 of 1 expected record(s)" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 1, + "distinct_locations": 1, + "location_is_seed": true, + "seed_creates": 1, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 15.5 + }, + { + "id": "repeat-create-merged", + "class": "no_new_record", + "contract": "register_patient.yaml", + "right_answer": "CONFIRMED", + "outcome": "CONFIRMED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "held", + "frame": "held", + "source": "instruction" + }, + "observed": "present", + "reasons": [], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 1, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 1, + "locations_in_search": 1 + }, + "seconds": 4.6 + }, + { + "id": "extra-record", + "class": "duplicate", + "contract": "register_patient.yaml", + "right_answer": "REFUTED", + "outcome": "REFUTED", + "right": true, + "fault_passed_as_verified": false, + "checked": { + "delta": "refuted", + "frame": "refuted", + "source": "instruction" + }, + "observed": "conflicting", + "reasons": [ + "exact_new_set was refuted: observed conflicting, 2 of 1 expected record(s)", + "1 record(s) outside the declared write moved (added); 1 of them no actor explains" + ], + "not_covered": [ + "writes that land after read_at (delayed or queued writes)", + "stores the contract doesn't read: email, other services, files", + "whether the write was authorized; the verifier doesn't judge it", + "records outside the read set (no frame.database)" + ], + "read_paths": [ + "fhir" + ], + "truth": { + "creates_answered": 2, + "distinct_locations": 2, + "location_is_seed": false, + "seed_creates": 0, + "search_matches": 2, + "locations_in_search": 2 + }, + "seconds": 14.0 + } + ] +} diff --git a/packages/flow/benchmark/commercial_ehr/epic/suite.py b/packages/flow/benchmark/commercial_ehr/epic/suite.py new file mode 100644 index 000000000..7f6891e10 --- /dev/null +++ b/packages/flow/benchmark/commercial_ehr/epic/suite.py @@ -0,0 +1,354 @@ +"""A small write-fault suite on Epic's public FHIR sandbox, through the verifier. + + set -a; source /path/to/.env; set +a + PYTHONPATH=. python -m benchmark.commercial_ehr.epic.suite --out results.json + PYTHONPATH=. python -m benchmark.commercial_ehr.epic.suite \\ + --suite SUITE.yaml --sha256 DIGEST --out results.json + +Each case registers one synthetic patient with a family name unique to the +run, the way an agent would: ``begin_write`` reads the sandbox, the "agent" +(this harness, with the app's backend credential) sends its FHIR create, and +``verify_write`` reads the sandbox again and judges the write against +``register_patient.yaml``. The right answer of each case is fixed before it +runs, from what the probe established about Epic (FINDINGS.md). + +The sandbox is shared and Epic may wipe it. Nothing is deleted (the app has +no delete scope), so each case writes at most three patients, and the whole +suite about a dozen. SSNs, which Epic requires on a create, are drawn from +the never-issued 000 area. The results hold case ids, outcomes and reasons, +no values and no credentials. +""" + +from __future__ import annotations + +import argparse +import json +import os +import random +import string +import tempfile +import time +from datetime import date, timedelta +from pathlib import Path +from typing import Any, Callable + +from benchmark.commercial_ehr.epic.client import EpicClient +from openadapt_flow.verifier import Actor, Contract, begin_write, verify_write + +HERE = Path(__file__).resolve().parent +SSN_SYSTEM = "urn:oid:2.16.840.1.113883.4.1" + + +def _ssn() -> str: + return "000-%02d-%04d" % (random.randint(10, 99), random.randint(1000, 9999)) + + +def _family() -> str: + return "Oasuite" + "".join(random.choice(string.ascii_lowercase) for _ in range(7)) + + +def task_values(phone: bool = False) -> dict[str, str]: + values = { + "family": _family(), + "given": random.choice(["Ada", "Grace", "Alan", "Edsger"]), + "birth_date": f"19{random.randint(50, 99)}-0{random.randint(1, 9)}-1{random.randint(0, 9)}", + "gender": random.choice(["female", "male"]), + "street": f"{random.randint(2, 99)} Test Way", + "city": "Boston", + "state": "MA", + "postal_code": "02110", + "email": "suite.probe@example.invalid", + } + if phone: + values["phone"] = "617-555-0142" + return values + + +def patient(v: dict[str, str], **override: str) -> dict[str, Any]: + v = {**v, **override} + telecom = [{"system": "email", "value": v["email"], "use": "home"}] + if "phone" in v: + telecom.insert(0, {"system": "phone", "value": v["phone"], "use": "home"}) + return { + "resourceType": "Patient", + "identifier": [{"use": "usual", "system": SSN_SYSTEM, "value": v.get("ssn") or _ssn()}], + "name": [{"use": "usual", "family": v["family"], "given": [v["given"]]}], + "gender": v["gender"], + "birthDate": v["birth_date"], + "telecom": telecom, + "address": [ + { + "use": "home", + "line": [v["street"]], + "city": v["city"], + "state": v["state"], + "postalCode": v["postal_code"], + "country": "US", + } + ], + } + + +def _create(c: EpicClient, body: dict[str, Any]) -> str: + status, headers, _ = c.post("Patient", body) + if status != 201: + raise RuntimeError(f"create returned HTTP {status}") + return headers.get("location", "") + + +def _day_after(iso: str) -> str: + return (date.fromisoformat(iso) + timedelta(days=1)).isoformat() + + +# Each case: (id, class, contract, phone in the task, seed, act, right answer). +# ``seed`` runs before the baseline; ``act`` is the agent's write. +Act = Callable[[EpicClient, dict[str, str], dict[str, Any]], None] +CASES: list[tuple[str, str, str, bool, Act, Act, str]] = [ + ("clean-1", "clean", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v))), + "CONFIRMED"), + ("clean-2", "clean", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v))), + "CONFIRMED"), + # Epic answers 201 and silently drops the phone (probe: every format). + ("phone-dropped", "silent_drop", "register_patient_phone.yaml", True, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v))), + "REFUTED"), + # The search is by the task's birth date, so a record saved under another + # one is outside the read set: nothing the task asked for is there, and + # the agent sent a write, so UNCERTAIN (round 1 expected REFUTED). + ("wrong-birth-date", "wrong_field", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v, birth_date=_day_after(v["birth_date"])))), + "not CONFIRMED"), + ("wrong-city", "wrong_field", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v, city="Cambridge"))), + "REFUTED"), + ("wrong-given", "wrong_field", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v, given="Zed"))), + "REFUTED"), + # The agent says it saved, and sent nothing. + ("no-write", "no_write", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: None, + "not CONFIRMED"), + # The person already exists (same SSN); Epic answers 201 with the old + # record's Location instead of creating one. + ("existing-returned-201", "no_new_record", "register_patient.yaml", False, + lambda c, v, s: s.update(seed=_create(c, patient(v, ssn=s.setdefault("ssn", _ssn())))), + lambda c, v, s: s.update(loc=_create(c, patient(v, ssn=s["ssn"]))), + "not CONFIRMED"), + # Two creates of the same person under different SSNs. Round 1 expected + # two records; Epic merged them into one (both creates answered the same + # Location), so the one record holding every value is right. + ("repeat-create-merged", "no_new_record", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update(loc=_create(c, patient(v)), loc2=_create(c, patient(v))), + "CONFIRMED"), + # A second create that differs only in the street is merged too (probed: + # one Location, the first street kept), so the extra record here differs + # in the given name: Epic keeps both, two records with the task's family. + ("extra-record", "duplicate", "register_patient.yaml", False, + lambda c, v, s: None, + lambda c, v, s: s.update( + loc=_create(c, patient(v)), loc2=_create(c, patient(v, given="Other")) + ), + "REFUTED"), +] + + +def ground_truth(c: EpicClient, v: dict[str, str], state: dict[str, Any]) -> dict[str, Any]: + """What the harness itself reads after the write: counts, never values. + + The same FHIR API the verifier reads, but by the Locations the creates + returned as well as by search, so a record the search can't see still + counts. + """ + + locs = [state[k] for k in ("loc", "loc2") if state.get(k)] + list(state.get("locs", [])) + seeds = [state["seed"]] if state.get("seed") else list(state.get("seeds", [])) + _, _, bundle = c.get("Patient", {"family": v["family"], "birthdate": v["birth_date"]}) + found = sorted( + e["resource"]["id"] + for e in (bundle or {}).get("entry", []) + if e.get("resource", {}).get("resourceType") == "Patient" + ) + return { + "creates_answered": len(locs), + "distinct_locations": len(set(locs)), + "location_is_seed": any(x in seeds for x in locs), + "seed_creates": len(seeds), + "search_matches": len(found), + "locations_in_search": sum(1 for x in set(locs) if x.split("/")[-1] in found), + } + + +# -- frozen suite files ----------------------------------------------------------- + +#: What a case may change from the task's values in one create. +OVERRIDES = { + "given", "gender", "street", "city", "state", "postal_code", "email", + "phone", "family_suffix", "birth_date_shift_days", "omit", "ssn", +} +RIGHT_ANSWERS = {"CONFIRMED", "REFUTED", "UNCERTAIN", "not CONFIRMED"} + + +def _apply(v: dict[str, str], o: dict[str, Any], state: dict[str, Any]) -> dict[str, Any]: + """One create's body: the task's values with the case's changes.""" + + bad = set(o) - OVERRIDES + if bad: + raise ValueError(f"unknown override(s): {sorted(bad)}") + values = {**v} + for key in ("given", "gender", "street", "city", "state", "postal_code", "email", "phone"): + if key in o: + values[key] = str(o[key]) + if "family_suffix" in o: + values["family"] = v["family"] + str(o["family_suffix"]).lower() + if "birth_date_shift_days" in o: + values["birth_date"] = ( + date.fromisoformat(v["birth_date"]) + timedelta(days=int(o["birth_date_shift_days"])) + ).isoformat() + ssn = o.get("ssn", "new") + if ssn == "same_as_seed": + values["ssn"] = state.setdefault("ssn", _ssn()) + elif ssn == "seed": + values["ssn"] = state["ssn"] = state.get("ssn") or _ssn() + elif ssn != "new": + raise ValueError("ssn is new, seed or same_as_seed") + body = patient(values) + for name in o.get("omit") or []: + if name == "email": + body["telecom"] = [x for x in body["telecom"] if x["system"] != "email"] + elif name == "phone": + body["telecom"] = [x for x in body["telecom"] if x["system"] != "phone"] + elif name in ("street", "city", "state", "postal_code"): + key = {"street": "line", "postal_code": "postalCode"}.get(name, name) + body["address"][0].pop(key, None) + elif name in ("gender", "birth_date"): + body.pop({"birth_date": "birthDate"}.get(name, name), None) + elif name == "given": + body["name"][0].pop("given", None) + else: + raise ValueError(f"can't omit {name}") + return body + + +def case_from_spec(spec: dict[str, Any]) -> tuple: + """A suite-file case as the tuple ``run_case`` takes.""" + + right = spec["right_answer"] + if right not in RIGHT_ANSWERS: + raise ValueError(f"{spec.get('id')}: right_answer {right!r}") + seeds = list(spec.get("seed") or []) + agent = list(spec.get("agent") or []) + if len(seeds) + len(agent) > 4: + raise ValueError(f"{spec['id']}: at most four creates a case") + + def seed(c: EpicClient, v: dict[str, str], s: dict[str, Any]) -> None: + for o in seeds: + s.setdefault("seeds", []).append(_create(c, _apply(v, o or {}, s))) + + def act(c: EpicClient, v: dict[str, str], s: dict[str, Any]) -> None: + for o in agent: + s.setdefault("locs", []).append(_create(c, _apply(v, o or {}, s))) + + contract = "register_patient_phone.yaml" if spec.get("task_phone") else "register_patient.yaml" + return (str(spec["id"]), str(spec.get("class", "")), contract, + bool(spec.get("task_phone")), seed, act, right) + + +def load_suite(path: Path, sha256: str) -> list[tuple]: + import hashlib + + import yaml + + raw = path.read_bytes() + digest = hashlib.sha256(raw).hexdigest() + if digest != sha256: + raise SystemExit(f"suite digest {digest} is not the frozen {sha256}") + data = yaml.safe_load(raw) + if data.get("schema") != "openadapt.epic-fault-suite/v1": + raise SystemExit("not an openadapt.epic-fault-suite/v1 file") + return [case_from_spec(case) for case in data["cases"]] + + +def run_case(c: EpicClient, case: tuple, workdir: Path) -> dict[str, Any]: + cid, klass, contract_file, phone, seed, act, right = case + v = task_values(phone) + state: dict[str, Any] = {} + seed(c, v, state) + os.environ["EPIC_TASK_FAMILY"] = v["family"] + os.environ["EPIC_TASK_BIRTHDATE"] = v["birth_date"] + contract = Contract.load(HERE / contract_file) + actor = Actor("agent", "epic-sandbox-suite") + started = time.monotonic() + pending = begin_write(contract, v, actor=actor, ledger=None, workdir=workdir) + act(c, v, state) + result = verify_write(pending, seal=False) + truth = ground_truth(c, v, state) + outcome = result.outcome.value if hasattr(result.outcome, "value") else str(result.outcome) + right_ok = outcome == right if right != "not CONFIRMED" else outcome != "CONFIRMED" + return { + "id": cid, + "class": klass, + "contract": contract_file, + "right_answer": right, + "outcome": outcome, + "right": right_ok, + "fault_passed_as_verified": outcome == "CONFIRMED" and right != "CONFIRMED", + "checked": {k: str(x) for k, x in result.checked.items()}, + "observed": result.observed, + "reasons": list(result.reasons), + "not_covered": list(result.scope.not_covered), + "read_paths": list(result.scope.read_paths), + "truth": truth, + "seconds": round(time.monotonic() - started, 1), + } + + +def main(argv: list[str] | None = None) -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--out", required=True) + ap.add_argument("--only", nargs="*") + ap.add_argument("--suite", type=Path, help="a frozen openadapt.epic-fault-suite/v1 file") + ap.add_argument("--sha256", help="the suite file's frozen digest") + args = ap.parse_args(argv) + cases = CASES + if args.suite is not None: + if not args.sha256: + raise SystemExit("--suite needs --sha256") + cases = load_suite(args.suite, args.sha256) + c = EpicClient() + rows = [] + with tempfile.TemporaryDirectory() as tmp: + for case in cases: + if args.only and case[0] not in args.only: + continue + try: + row = run_case(c, case, Path(tmp)) + except Exception as exc: # noqa: BLE001 - a harness error is reported, not judged + row = {"id": case[0], "class": case[1], "error": f"{type(exc).__name__}: {exc}"[:300]} + rows.append(row) + print(json.dumps({k: row.get(k) for k in ("id", "right_answer", "outcome", "right", "error")})) + summary = { + "cases": len(rows), + "right": sum(1 for r in rows if r.get("right")), + "faults_passed_as_verified": sum(1 for r in rows if r.get("fault_passed_as_verified")), + "harness_errors": sum(1 for r in rows if "error" in r), + "sandbox_calls": c.calls, + "date": time.strftime("%Y-%m-%d", time.gmtime()), + "suite_sha256": args.sha256 or "built-in", + } + Path(args.out).write_text(json.dumps({"summary": summary, "cases": rows}, indent=1) + "\n") + print(json.dumps(summary)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/openadapt_flow/ensure_record.py b/packages/flow/openadapt_flow/ensure_record.py index a892078e6..edb1e5054 100644 --- a/packages/flow/openadapt_flow/ensure_record.py +++ b/packages/flow/openadapt_flow/ensure_record.py @@ -458,6 +458,21 @@ def bundle( if not isinstance(page, list): raise Unreadable(f"{path}: entry is not a list") entries += [e for e in page if isinstance(e, dict)] + for e in page: + res = e.get("resource") if isinstance(e, dict) else None + if (e.get("search") or {}).get("mode") != "outcome" or not isinstance( + res, dict + ): + continue + # A warning ("no results") is a message; an error means the + # search may be partial, and a missed match is a duplicate. + issues = res.get("issue") or [] + if not isinstance(issues, list) or any( + not isinstance(i, dict) + or i.get("severity") not in ("warning", "information") + for i in issues + ): + raise Unreadable(f"{path}: the search returned an error outcome") nxt = next( ( link.get("url") diff --git a/packages/flow/openadapt_flow/runtime/effects/fhir.py b/packages/flow/openadapt_flow/runtime/effects/fhir.py index 39a4bf7e6..2384806e9 100644 --- a/packages/flow/openadapt_flow/runtime/effects/fhir.py +++ b/packages/flow/openadapt_flow/runtime/effects/fhir.py @@ -380,7 +380,22 @@ def get(target: str, params: Optional[dict[str, str]]) -> Any: self.read_problem = "a Bundle entry has no resource" return None mode = (entry.get("search") or {}).get("mode") - if mode not in ("include", "outcome"): + if mode == "outcome" or ( + mode is None and resource.get("resourceType") == "OperationOutcome" + ): + # A message about the search, not a record (Epic sends a + # warning "returns no results" on an empty search). An + # error in it means the set may be partial. + issues = resource.get("issue") or [] + if not isinstance(issues, list) or any( + not isinstance(i, dict) + or i.get("severity") not in ("warning", "information") + for i in issues + ): + self.read_problem = "the search returned an error outcome" + return None + continue + if mode != "include": matches += 1 out.append(self._flatten(resource)) next_url = _next_link(bundle) diff --git a/packages/flow/public-artifacts.json b/packages/flow/public-artifacts.json index 1b146f912..70254636c 100644 --- a/packages/flow/public-artifacts.json +++ b/packages/flow/public-artifacts.json @@ -111,6 +111,14 @@ "benchmark/code_as_action/results/openemr/summary.json", "benchmark/code_as_action/results/openemr/superseded.jsonl", "benchmark/code_as_action/results/spend.json", + "benchmark/commercial_ehr/epic/blind_summary.json", + "benchmark/commercial_ehr/epic/probe.json", + "benchmark/commercial_ehr/epic/register_patient.yaml", + "benchmark/commercial_ehr/epic/register_patient_phone.yaml", + "benchmark/commercial_ehr/epic/round1.json", + "benchmark/commercial_ehr/epic/round1_followups.json", + "benchmark/commercial_ehr/epic/round2.json", + "benchmark/commercial_ehr/epic/round3.json", "benchmark/comparison_artifact/comparison.html", "benchmark/comparison_artifact/comparison.json", "benchmark/compile_agent/results/identical-inputs/runs.jsonl", diff --git a/packages/flow/tests/test_effect_paging.py b/packages/flow/tests/test_effect_paging.py index 8f7740a0c..e79e9c2fd 100644 --- a/packages/flow/tests/test_effect_paging.py +++ b/packages/flow/tests/test_effect_paging.py @@ -310,6 +310,66 @@ def page(path: str, n: int) -> tuple: assert len(_read(_fhir(holder["url"])) or []) == 6 +def _empty_with_outcome(severity: str) -> dict: + """Epic's empty search: no matches and an OperationOutcome entry.""" + + return { + "resourceType": "Bundle", + "type": "searchset", + "total": 0, + "entry": [ + { + "resource": { + "resourceType": "OperationOutcome", + "issue": [{"severity": severity, "code": "processing"}], + }, + "search": {"mode": "outcome"}, + } + ], + } + + +def test_fhir_a_warning_outcome_is_not_a_record(serve) -> None: + """Counted as a record, the outcome vanished once the patient was + created, and the frame refused a clean write as a removed record.""" + + url = serve(lambda path, n: (200, _empty_with_outcome("warning"))) + assert _read(_fhir(url)) == [] + + +def test_fhir_an_error_outcome_is_unreadable(serve) -> None: + url = serve(lambda path, n: (200, _empty_with_outcome("error"))) + state = _fhir(url).capture_pre_state() + assert not state.reachable + assert state.detail["read_problem"] == "the search returned an error outcome" + + +def test_fhir_an_outcome_without_a_mode_is_not_a_record(serve) -> None: + def page(path: str, n: int) -> tuple: + bundle = _empty_with_outcome("information") + bundle["entry"][0].pop("search") + return 200, bundle + + assert _read(_fhir(serve(page))) == [] + + +def test_fhir_an_outcome_beside_matches_on_a_later_page(serve) -> None: + """Matches, a warning outcome on page two, and a total that counts only + the matches: every match is read, the outcome is not.""" + + holder: dict = {} + + def page(path: str, n: int) -> tuple: + bundle = _bundle(holder["url"], n, per=3, total=5) + if n == 1: + bundle["entry"] += _empty_with_outcome("warning")["entry"] + return 200, bundle + + holder["url"] = serve(page) + records = _read(_fhir(holder["url"])) + assert [r["id"] for r in records or []] == ["1", "2", "3", "4", "5"] + + def test_fhir_next_origin_is_named(serve) -> None: url = serve(lambda path, n: (200, _bundle("http://203.0.113.9", n))) state = _fhir(url).capture_pre_state() diff --git a/packages/flow/tests/test_ensure_record.py b/packages/flow/tests/test_ensure_record.py index a9c4a5769..500ae148b 100644 --- a/packages/flow/tests/test_ensure_record.py +++ b/packages/flow/tests/test_ensure_record.py @@ -183,6 +183,19 @@ def bundle(*resources, total=None, next_url=None, kind="searchset"): return body +def with_outcome(body, severity): + body["entry"] = list(body.get("entry") or []) + [ + { + "resource": { + "resourceType": "OperationOutcome", + "issue": [{"severity": severity, "code": "processing"}], + }, + "search": {"mode": "outcome"}, + } + ] + return body + + def reader_for(routes): def get(url, params): key = url.split("/fhir/R4/", 1)[-1] @@ -262,6 +275,7 @@ def test_log_and_record_disagreeing_on_the_author_is_unattributed(): {"AuditEvent": bundle(next_url="http://elsewhere.test/fhir/R4/AuditEvent?p=2")}, {"AuditEvent": bundle({"resourceType": "AuditEvent"})}, {"AuditEvent": bundle(audit("C", None))}, + {"AuditEvent": with_outcome(bundle(), "error")}, ], ids=[ "http-500", @@ -270,6 +284,7 @@ def test_log_and_record_disagreeing_on_the_author_is_unattributed(): "other-origin", "no-id", "no-entity", + "error-outcome", ], ) def test_every_incomplete_read_is_unreadable(routes): @@ -280,6 +295,18 @@ def test_every_incomplete_read_is_unreadable(routes): ) +def test_a_warning_outcome_is_a_message_not_an_unreadable_search(): + """Epic answers an empty search with a warning outcome entry.""" + + r = reader_for( + {"Observation": bundle(), "AuditEvent": with_outcome(bundle(), "warning")} + ) + changes = read_changes( + contract(change_feed={"audit": {}, "sweep_types": ["Observation"]}), r, T0 + ) + assert changes == {} + + def test_a_server_without_history_authors_relies_on_the_audit_trail(): feed = {"audit": {}, "sweep_types": ["Observation"], "history_author": False} r = reader_for(