Skip to content

benchmark(commercial_ehr): qualify the verifier on Epic's public FHIR sandbox - #1656

Merged
abrichr merged 6 commits into
mainfrom
research/epic-sandbox-verifier
Oct 29, 2026
Merged

abrichr merged 6 commits into
mainfrom
research/epic-sandbox-verifier

Conversation

@abrichr

@abrichr abrichr commented Oct 29, 2026 •

Copy link
Copy Markdown
Member

Phase 2 of qualifying the verifier against a commercial EHR through a public developer sandbox; Phase 1 (the terms) merged in #1650. No agent created an account; the founder registered the Epic app.

FINDINGS.md gains a Phase 2 section, and its registration steps now match what Epic asked for (a JWK Set URL).

Phase 2, Epic (benchmark/commercial_ehr/epic/):

  • Read path, probed (probe.json): search and read work. There is no AuditEvent and no _history (both 404) and no meta. _lastUpdated is refused on Patient and silently ignored on Observation. Provenance is available only through _revinclude.
  • Epic's write behavior:
    • A Patient create answers 201 but drops the phone (email kept).
    • A repeat create of an existing person answers 201 with the old record's Location.
    • A created allergy doesn't appear in search.
    • An Observation create needs an Encounter, which the app has no scope to find.
  • Verifier: openadapt_flow.verifier with the fhir record check and its existing smart_backend auth, contract register_patient.yaml. The frame is the search set. The claim names the path: read back through Epic's FHIR search and read.
  • Blind round: 13 cases from a separate agent, frozen by sha256 (ff402d6a…c08d) at commit c93069d0d and run once. 12/13 right, none passed as verified. The miss was a false refusal of a clean save beside a look-alike patient. Each value check matched every record with the task's family name, which was a contract defect. The contract now uses one exact_new_set effect. Only the hash and counts are public (blind_summary.json).
  • Non-blind rounds: 7/9, then 10/10, then 10/10 after the contract fix. A rerun of the blind file after the fix got 13/13; it is no longer blind.
  • Read path, named in RESULTS.md: FHIR search and read only. There is no AuditEvent and no _history. The probe read Provenance through _revinclude, but the verifier doesn't use it, so authorship isn't checked.

Fix found by the sandbox: runtime/effects/fhir.py counted a search's OperationOutcome entry (Epic's empty-search warning) as a record, so a clean create read as a removed record and was refused. A warning or information outcome is now skipped, and an error outcome makes the read unreadable. An OperationOutcome with no search.mode is handled the same way, and ensure_record now treats an error outcome as unreadable. The adversarial review passed with no blocking finding (REVIEW.md); tests cover each case.

Also: a CAPABILITIES.md row, and a regenerated llms-full.txt. Tests run: test_effect_paging, the FHIR, verifier and ensure_record tests, the capabilities map and shipped-copy tests (470 passed).

…c FHIR sandbox

Phase 2: probe Epic's read path, contract it for openadapt_flow.verifier
with SMART backend auth, and run a write-fault suite (round 2: 10/10,
none passed as verified). Epic drops a created patient's phone on a 201,
answers 201 with an existing record's Location for a repeat person, and
hides created allergies from search.

Fix: the FHIR record check counted a search's OperationOutcome entry as a
record, so a clean create on Epic read as a removed record. A warning or
information outcome is now skipped; an error outcome is unreadable.
…handling

A blind 13-case suite (sha256 ff402d6a...) got 12/13 with no fault passed
as verified. The miss was a false refusal of a clean save beside a
look-alike patient: each value check matched every record with the task's
family name. The contract now declares one exact_new_set effect.

Adversarial review of the FHIR outcome fix passed with no blocking
finding. From its findings: an OperationOutcome with no search.mode is
an outcome too, and ensure_record's reader now treats an error outcome as
unreadable instead of as no match.
…-verifier

# Conflicts:
#	CAPABILITIES.md
#	llms-full.txt
@abrichr
abrichr enabled auto-merge (squash) October 29, 2026 17:36
@abrichr
abrichr merged commit c3cc84d into main Oct 29, 2026
45 checks passed
@abrichr
abrichr deleted the research/epic-sandbox-verifier branch October 29, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant