[abrichr]

Adds a pre-registered leaderboard of computer-use agents on the 37 public Medplum cases, run in Medplum's own web app.

What is in it (packages/flow/benchmark/fhir_medplum/leaderboard/):

• PROTOCOL.md: the protocol, frozen before any scored run. Its SHA-256, and the hashes of every harness file, are committed to openadapt-internal (evidence/write-leaderboard-2026-11/PREREGISTRATION.sha256).
• browser.py, run.py: the harness signs in to the Medplum web app, routes the app's FHIR calls through the public fault proxy inside the browser, and scores each run with public/oracle.py. Anyone can run their own agent with --agent py:<module>:<function>.
• agents/: adapters for Claude computer use, OpenAI's computer tool, Gemini's computer use (built, not run: no API credit) and Fara-7B on MLX. The Fara prompt is vendored MIT text from microsoft/fara.
• score.py: the three metrics (said done with the record wrong, collateral writes said done, pass^k with k=3), bootstrap intervals over cases.
• REPORT.md, DEVIATIONS.md, results/: every scored row with its transcript and first and last screenshots, the excluded signed-out runs, and the spend ledger.

Results (3 runs of each case):

Agent	Said done, record wrong	Collateral writes said done	pass^3
claude-sonnet-5-5	25.2% (CI 13.5–37.8)	20 of 20	56.8%
gpt-6.1-sol	29.7% (CI 16.2–46.0)	30 of 30	62.2%

The intervals overlap; the data do not rank the two. Neither misreported a clean case. Fara-7B's rows are still running locally and will be added by a follow-up commit.

Three deviations are logged with reasons: a host-load guard, Fara run one process at a time, and 188 runs rerun after the app's one-hour session expired under them.

Spend at list prices: $64.78 (claude) and $20.48 (openai), including the excluded runs.

[abrichr]

Correction (841d1e7, deviation 4): the verifier's read token expired an hour into each process, so the live shadow column judged only 59 of 222 runs. The shadow column now uses verdicts replayed from Medplum's stored history (#1795): claude flagged 25 of 28 misreports with 26 of 72 false flags, and openai 30 of 33 with 11 of 68. The end-state check uses a separately renewed token, so the three ranked metrics are unchanged. public/run.py now renews the verifier's token before each baseline and each check.

[abrichr]

From: Grok, To: coordinator

APPROVE at 99e3860. Checked in a detached worktree with main merged in locally (clean merge, not pushed).

Checked:

• Scores reproduce. python -m benchmark.fhir_medplum.leaderboard.score regenerates LEADERBOARD.json and LEADERBOARD.md byte for byte, with no diff. Claude: 25% said done with the record wrong (CI 14–38), pass^3 57%. OpenAI: 30% (CI 16–46), pass^3 62%. Both had 111 runs.
• Packaging. results/ is excluded from both the wheel and the sdist, and it is listed in public-artifacts and in the approved directories of check_release_consistency.py. The script passes. test_release_contract, test_release_publication, test_bench_medplum_public and test_release_ci_gate: 119 passed. CI is green.
• Privacy scan of the full diff (1,912 files, 44 MB of text):
  • no API keys, JWTs, bearer tokens or AWS account ids;
  • only example.org synthetic emails;
  • no private IPs.
  • The sign-in credentials come from the fixture's generated state and never reach results/.

P2:

• The pre-registration can't be checked from outside. The protocol hash and harness hashes are only in a private repo, so outside readers can't verify the "frozen before any scored run" claim. Publishing the hash file itself (hashes only) next to PROTOCOL.md would fix that.
• The PR body is out of date on Fara. It says Fara's rows are still running, but results/fara already has 57 runs (the first half), and LEADERBOARD.md shows them: 26% said done wrong, pass^3 0%. The body and REPORT should say this row covers half the cases.

P3:

• Ruff errors in leaderboard/file_times.py (E401, I001, E702). CI doesn't lint this path.
• Local paths in the committed tracebacks. Two tracebacks in results/ include a local home-directory worktree path. That's harmless, but they could be trimmed.
• CAPABILITIES says "On main" before merge. That's the usual pattern, but it's only true once merged.
• No unit tests for the new code. score.py and common.py have none; the reproduced board is the only check.

[abrichr]

Independent adversarial review: BLOCK

Reviewed-Head: 1542d08

Reviewed-Parent: 15ad55b

The intended author delta is from merge-base e67d1e0 to this head. It contains the leaderboard, its results, packaging/inventory entries and two capability rows. The literal parent already has identical leaderboard code; these are shared defects, not regressions introduced by the final merge.

• P2 — Preserve sampled case units in pass^3 intervals. In packages/flow/benchmark/fhir_medplum/leaderboard/score.py:167, bootstrap samples flatten repeated case draws, then pass_hat_k at line 113 merges those copies by their original ID. A case with two passes and one failure contributes 0 to pass^3, but sampling it twice becomes four passes in six trials and contributes 0.2. An independent two-case control where neither case passes all three runs produces CI [0.0, 0.2] instead of [0.0, 0.0] on both head and parent. Preserve each sampled occurrence as its own case unit, or resample the per-case statistic, and regenerate the intervals from the retained rows.
• P2 — Enforce the app boundary on actual browser requests. packages/flow/benchmark/fhir_medplum/leaderboard/browser.py:265 intercepts only FHIR traffic; it does not enforce the protocol's refusal of off-app actions. Actual OpenAI and Claude adapter clicks on a link in an owned headless browser sent a request to a second owned loopback origin on both revisions. The same-app controls passed. The evidence proves an off-origin request, not that the destination document finished loading. Add a browser-level boundary covering navigation from clicks, keyboard actions and direct navigation; a check only in the explicit URL tools does not cover this path.

Validation: all 134 existing tests in the complete release-contract, source-policy-binding and public Medplum modules passed. The first combined run was 143 passed / 7 failed in 65.48 seconds. A bounded navigation follow-up corrected an observation race without changing its assertions: 2 same-app controls passed and all 4 head/parent adapter confinement controls failed in 12.11 seconds. Bootstrap all-pass/all-fail, claim and infrastructure-exclusion controls passed. The committed board JSON and Markdown reproduce byte for byte. Source-boundary and release-consistency checks passed. Focused Ruff reports the three existing file_times.py errors E401/I001/E702.

Limits: this is code/scoring review, with synthetic offline and owned-loopback probes. No model calls, live app fixture, external destination or native UI was used. PNGs were not visually or OCR audited; the retained result artifacts are not privacy-certified here. No wheel/sdist was built. At the inspected head, Python CodeQL failed in the upload phase with no retained explicit cause, so CI is not described as fully green. Website, paper and outreach approval are outside this review.

[abrichr]

APPROVE — bootstrap sampling and HTTP navigation delta only

From: Codex
Reviewed-Head: 5ce4fc4

The sampler gives every draw its own case ID before statistics group its repetitions, preserving independent case units. All four head scoring controls passed, including the mixed-case example that must never invent an all-pass case.

The whole author module recorded nine passes and its original missing-board setup skip. The genuine click, keyboard, popup and script-navigation cases recorded blocked navigation and zero requests at the owned outside HTTP server; direct off-app goto refused and same-app navigation worked. The three private browser controls also passed, but their adapter cases did not retain an attempted-navigation callback, so they receive no extra attempted-navigation credit. This validates the measured HTTP navigation boundary; subresources and arbitrary protocols are outside that claim.

The overall run remains incomplete: 22 collected cases, 65 phases, 19 passes, two assertion failures and one setup skip. One failure is the retained head diagnostic that manually pools duplicate IDs while bypassing the corrected sampler. Parent scoring recorded three passes and the mixed-case failure, then its post-test source-origin check raised before the after snapshot; parent browser/diagnostic groups and the final integrity footer were not reached. Complete parent attribution and full-harness qualification remain unestablished.

I independently reconciled phase/JUnit/checkpoint records and rehashed 236 loaded public ref/path pairs against Git. Required Ruff, private formatting and Python 3.10 syntax checks passed. Immutable public-format diagnostics exited nonzero on both refs and remain recorded. Mypy and actual Python 3.10 execution were not run. No original assertion, skip or refusal gate was weakened.

Generality: no per-app engine change; one draw-unit model and context-level HTTP navigation rule cover the reviewed entry points. Held-out evidence: synthetic software only. This approval does not qualify committed board results, model runs, native applications or real-app/M1 acceptance.

[abrichr]

Codex delta review: APPROVE at 404f6fe against the earlier approved 5ce4fc4. The only change is the one-line allow_abbrev=False in benchmark/fhir_medplum/leaderboard/run.py.