Repository navigation
benchmark(commercial_ehr): qualify the verifier on Epic's public FHIR sandbox - #1656
Merged
Merged
Conversation
…c FHIR sandbox Phase 2: probe Epic's read path, contract it for openadapt_flow.verifier with SMART backend auth, and run a write-fault suite (round 2: 10/10, none passed as verified). Epic drops a created patient's phone on a 201, answers 201 with an existing record's Location for a repeat person, and hides created allergies from search. Fix: the FHIR record check counted a search's OperationOutcome entry as a record, so a clean create on Epic read as a removed record. A warning or information outcome is now skipped; an error outcome is unreadable.
…handling A blind 13-case suite (sha256 ff402d6a...) got 12/13 with no fault passed as verified. The miss was a false refusal of a clean save beside a look-alike patient: each value check matched every record with the task's family name. The contract now declares one exact_new_set effect. Adversarial review of the FHIR outcome fix passed with no blocking finding. From its findings: an OperationOutcome with no search.mode is an outcome too, and ensure_record's reader now treats an error outcome as unreadable instead of as no match.
…-verifier # Conflicts: # CAPABILITIES.md # llms-full.txt
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of qualifying the verifier against a commercial EHR through a public developer sandbox; Phase 1 (the terms) merged in #1650. No agent created an account; the founder registered the Epic app.
FINDINGS.md gains a Phase 2 section, and its registration steps now match what Epic asked for (a JWK Set URL).
Phase 2, Epic (
benchmark/commercial_ehr/epic/):probe.json): search and read work. There is no AuditEvent and no_history(both 404) and nometa._lastUpdatedis refused on Patient and silently ignored on Observation. Provenance is available only through_revinclude.openadapt_flow.verifierwith thefhirrecord check and its existingsmart_backendauth, contractregister_patient.yaml. The frame is the search set. The claim names the path: read back through Epic's FHIR search and read.ff402d6a…c08d) at commitc93069d0dand run once. 12/13 right, none passed as verified. The miss was a false refusal of a clean save beside a look-alike patient. Each value check matched every record with the task's family name, which was a contract defect. The contract now uses oneexact_new_seteffect. Only the hash and counts are public (blind_summary.json)._history. The probe read Provenance through_revinclude, but the verifier doesn't use it, so authorship isn't checked.Fix found by the sandbox:
runtime/effects/fhir.pycounted a search's OperationOutcome entry (Epic's empty-search warning) as a record, so a clean create read as a removed record and was refused. A warning or information outcome is now skipped, and an error outcome makes the read unreadable. An OperationOutcome with nosearch.modeis handled the same way, andensure_recordnow treats an error outcome as unreadable. The adversarial review passed with no blocking finding (REVIEW.md); tests cover each case.Also: a CAPABILITIES.md row, and a regenerated
llms-full.txt. Tests run:test_effect_paging, the FHIR, verifier and ensure_record tests, the capabilities map and shipped-copy tests (470 passed).