Repository navigation
Leaderboard re-score: replay Fara-7B's first half - #1837
Merged
Merged
Conversation
…ecked before the token fix
…ses confirmed; fidelity 4/4
Member
Author
Independent adversarial review: APPROVE, no blockersA separate agent with no part in writing this change reviewed head e59fbd0 read-only. Claims checked
Minor items and responses
|
abrichr
added a commit
that referenced
this pull request
Nov 4, 2026
abrichr
added a commit
that referenced
this pull request
Nov 9, 2026
…stered (#1737) * Write leaderboard harness: computer-use agents in the Medplum app, scored by the end-state oracle * Leaderboard: own-agent hook, pinned proxy port, README, protocol counts * Leaderboard protocol v1: limits from practice runs, per-agent shares, Gemini and Nova Act not run * Leaderboard: wait for host load under 35 before each run, record it; deviations log * Leaderboard: renew the Medplum session before each run; rerun signed-out runs (deviation 3) * Leaderboard deviations: time of deviation 3 * Leaderboard results: openai, all 111 scored runs, evidence and excluded signed-out runs * Leaderboard results: claude, all 111 scored runs, evidence and excluded signed-out runs * Leaderboard: results report for claude and openai; scored table, spend ledger, fara rows so far * CAPABILITIES: write leaderboard row; refresh llms-full * CAPABILITIES: leaderboard status * Flow packaging: register the leaderboard results folder as published evidence and keep it out of the wheel and sdist * Leaderboard: renew the verifier's token; replay the shadow column (deviation 4) The verifier's read token expired an hour into each process, so its live verdicts on 92 claude and 71 openai rows were unreadable. The end-state check uses a renewed admin token, so the three metrics are unchanged. The shadow column now uses verdicts replayed from Medplum's stored history (verifier-specificity, #1795): flagged 25/28 and 30/33, false flags 26/72 and 11/68. * Leaderboard: fara rows so far and their file times, for the shadow replay * Leaderboard: fara first half (P01-A06) done; its Medplum history and file times for the shadow replay * Leaderboard: deviation 2 notes fara's second process ran on 9820 (disk gate) * Leaderboard: fara first-half shadow verdicts replayed (#1837) * Leaderboard: Fara-7B scored (110 runs); report, CAPABILITIES Said done with the record wrong in 18.2% (CI 10.0-27.0), 20 of 22 claims wrong; 84 runs reached the 40-minute limit at a median 46 s per local model call. r2 A08 excluded (harness failed it three times while the database recovered). Five runs that overlapped a database restart are kept and listed in results/fara/db-restarts.json. * fix(leaderboard): count each bootstrap draw as its own case; block off-app navigation score.py pooled the runs of a case drawn twice under one id, so pass^3 intervals counted a 2-of-3 case drawn twice as 4 of 6. Each draw now gets its own id. Regenerated LEADERBOARD.json/md: point estimates unchanged, pass^3 intervals claude 45.8-69.6 -> 40.5-73.0, openai 50.0-73.9 -> 46.0-78.4, fara 0.0-1.1 -> 0.0-0.0. browser.py now aborts any navigation to another origin (click, key, script, pop-up, goto) inside the browser and logs it; Screen.goto refuses other origins. Deviations 5 and 6 record both, with what the retained evidence shows. Fix ruff errors in file_times.py. Add tests/test_write_leaderboard.py. * fix: disable argparse abbreviations in the Medplum leaderboard runner
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #1795. Fara-7B's first half (57 rows, P01–A06, stack 9820) ran before the verifier's token fix, so most of its live checks failed with HTTP 401. This PR replays them on the frozen server state, the same way #1795 replays claude and openai. It runs no new agent.
Pre-registered first.
PREREGISTRATION-FARA.mdandSHA256SUMS-FARAwere committed (22429e7) before any Fara verdict was computed. Every pinned file is unchanged in the results commit and on current main.Changes
inputs/frozen-9820.jsonl.gzandinputs/rows-fara.jsonl.gz, from Write leaderboard: computer-use agents in Medplum's web app, pre-registered #1737 at 01854c3; each row keeps its source path and sha256. Fara's times are merged intoinputs/times.json.fara: 9820stack, plusCHECKED_BEFORE, which skips Fara rows checked at or after 2026-11-04 11:55 UTC (those have live verdicts). The default agents are still claude and openai.--agent; the default is still the pre-registered pair.results/*-fara.json, plus a section in RESULTS.md and the inventory entries. The claude and openai results are not re-run.Public-data check: the frozen rows and dumps hold only synthetic bench-project records, example.org and .invalid emails, and no tokens or secrets.