[abrichr]

What

Code as action is now the default agent for a task's first run, and every save in that run is read back from the system of record before the agent goes on. A successful run compiles into a program as before.

• Default agent. ClaudeAgent() and openadapt-flow compile-agent now default to vision="code" on claude-sonnet-5-5, the setup scoreboard arm 7 measured. selective and every_turn remain available; selective and every_turn keep claude-opus-5-5.

• Read-back on every save the gate detects. After each action the save gate lets through, the harness reads the verify.yaml system of record and compares it with the read taken at the start of run 1 and with a read just before the save (harness.RecordReadBack, in both harnesses):

  • only a record created during run 1 can be the named one; a new record the task does not name, or any change to or removal of a starting record, stops run 1 (readback_wrong_record) after the wrong write has landed, and nothing can be admitted;
  • a second named record stops run 1 as a duplicate, counted even without record ids;
  • no new record is reported as "unconfirmed ... Do not save again", and the next save is refused if the first landed late;
  • a failed read stops run 1, and run 1 does not start without a starting read;
  • a save that raises is still read back;
  • landed requires every input value in the new record.

  Every read-back is kept in run1["readbacks"]. The pipeline always wires it.

• Teach path. A new MCP tool, teach_run (web apps), lets the client's model send action scripts. When a person approves saves (--teach-hold), the script stops before each save with needs_person, and the client's form asks the person. The approval is bound to that line on the app as it is at that moment; a stale approval does nothing. teach_act is unchanged.

• Recorder. After a save the gate let through, the recorder's after-read now waits for the system of record to change (Recorder.expect_write), so the compiler binds a late write to its save step. On OpenEMR the chart lands after the screen settles.

Measured (scoreboard, python -m scoreboard.build; PROTOCOL Amendments 10 to 12)

MockMed items 21 to 40 (20 items: item 30 is a save the server drops while the screen says saved, item 40 a save that lands on another patient). claude-sonnet-5-5. All rows are untimed because the host's load was above 12.

final attempt	Agent first runs (code, read-back)	Compiled program (from item 21)
Normal items correct	18 / 18	18 / 18
Item 30, dropped save	silent_wrong by the word rule; a safe stop on hand review	safe_stop
Item 40, wrong patient	flagged_wrong (the save landed on the other patient; the read-back then stopped run 1)	flagged_wrong
Silent wrong saves	1 by the scoreboard's word rule, 0 on hand review	0
Cost per item	$0.0093	$0
Model calls per item (median)	4	0
Task seconds, correct items (median)	6.6	5.3

On item 30 the agent said the save "did not take effect" because the read-back showed nothing was saved, but the scoreboard's word rule missed that wording. The row keeps the rule's verdict, and HAND_REVIEW.md records the review. The read-back's rules changed after review; the measured runs used the earlier rules, and by the new rules their verdicts are the same (Amendment 13; not re-run). From Amendment 11 on, arm 7 reports success only when its last read-back landed.

Real app (OpenEMR referral, local fixture). 6 of 6 correct by SQL read-back, and each save was read back as landed on the named chart. Admission still refuses these programs. The recorder fix binds the confirm save, but the "Create New Patient" click before it is write-shaped and writes nothing, so it has no read-back (Amendment 12). Fixing that changes what admission accepts, so it is not in this PR.

Spend: $0.43 on the scoreboard key (scoreboard total $12.98 of $30).

Tests

• tests/e2e/test_code_agent_default_e2e.py, MockMed in a real browser, scripted agents, no model calls, added to the flow-critical CI job:
  • read-back on every save;
  • a wrong-patient save stopped at the read-back;
  • a dropped save reported as not saved;
  • teach_run drafting a program;
  • teach_run stopping before a save and taking exactly one bound approval.
• Unit tests for record_readback, the new defaults, and the flags.
• 437 tests pass locally across the code-action, compile-agent, teach, JIT, authoring and recorder suites.

🤖 Generated with Claude Code

[abrichr]

AI review (coordinator)

Verdict: fix first. The direction is right and the e2e tests are real: MockMed in a real browser against the fault server, with no fakes standing in for the system of record. But the read-back accepts several writes it should refuse, and it can tell the agent "nothing was saved" when something was. Line numbers below are at fe7df6e91.

I checked RecordReadBack directly (unit probe, no browser) at this head:

case	outcome today	should be
a record that matches expect and existed before run 1 is overwritten	landed	refused
that record is deleted and a new matching one created	landed	refused
a second save creates a duplicate of the named record (keyed store)	landed	refused / held
the same duplicate in a store without an id field	no_change ("nothing was saved")	refused
mark() fails, then the second save is dropped	landed (the first save's write is counted)	unreadable, stop
the named record lands with an empty note	landed	landed_incomplete or similar

Tests run (serially, no xdist, host load 15 to 29): tests/test_code_action.py and tests/test_compile_agent.py pass (108 tests). All 5 tests in tests/e2e/test_code_agent_default_e2e.py pass, but test_teach_run_is_offered_and_drafts_a_program failed once under load with run 1 did not start: the app did not open in time and passed on rerun. This file is now in the flow-critical job, so expect that flake in the merge queue.

Blocking

1. A record that existed before run 1 counts as "the named record", so overwriting or deleting it reads back as landed. packages/flow/openadapt_flow/compile_agent/harness.py:127-144 (_others). named() matches only the expect identity fields (for example patient_id=p1, type=Triage). A change whose before and after both match, or a removal paired with any matching create, is allowed. On a real chart that already has a Triage record, run 1 can overwrite or replace clinical data, and the agent is told the save landed. check_run_record refuses it later, but by then the write is already in the EMR.
   Fix: for a create task, the named record is a record created during run 1 (its id or whole value is not in baseline()). Any change to or removal of a record that is in the baseline is wrong_record (or a new overwrote_existing), and it stops run 1. Add an e2e test that seeds a matching record before run 1.

2. A duplicate save is never caught, and in a store without an id field it reads back as "nothing was saved". harness.py:146-170. _others allows any number of named records to be created, so a second save on the named record reads back as landed. With no id field, diff_snapshots (record_sor.py:238-244) compares sets of serialized records, so an identical second row disappears. The agent is then told "nothing was saved ... do not report it as saved", which invites a third attempt.
   Fix: compare multisets (collections.Counter of _whole(r)) in the no-id path. In RecordReadBack, stop run 1 when more than one named record exists since baseline (duplicate), the same way wrong_record stops it.

3. A save that was sent but not confirmed can be retried blindly. harness.py:222 (readback_wait_s = 3.0, lines 337 and 778) and the no_change message at code_action.py:1503. A write that lands after 3 s reads back as no_change. The message says only "do not report it as saved", and nothing stops another save. Neither _CODE_SYSTEM (agents.py) nor the teach_run description tells the agent not to save again. In item 30 the agent happened to stop. Nothing enforces that.
   Fix: after a no_change or unreadable on a save, refuse the next gated commit unless a fresh read still matches the read taken at the no_change. If the store changed in between (a late landing), stop run 1 with RECONCILIATION_REQUIRED-style wording. Change the message to "the save is unconfirmed; do not save again; finish and report it as not confirmed".

4. A save whose dispatch raised is not read back. harness.py:470-481 and harness.py:891-899. On computer.execute raising (for example a Playwright timeout on a submit that navigates, which is common), the except BaseException re-raises before _read_back. In code_action a LookupError/UnsupportedAction then becomes "error: ... (a write may have been committed)" and the agent can resend the line. That is the "sent but not confirmed" case with no proof at all.
   Fix: when commit is not None, run _read_back in a finally (catching its own errors), log the outcome, and surface it in the error text. Treat no_change there like item 3 (no retry without a fresh matching read).

5. An unreadable system of record lets run 1 carry on. harness.py:150-151 returns unreadable, and _read_back (harness.py:213-235) only stops on wrong_record. If the baseline read failed (self._baseline is None, pipeline.py:513), every read-back is unreadable and the agent can make any number of saves with no wrong-record check. The PR says "every save ... is read back ... before the agent goes on". Fail closed instead.
   Fix: raise ReadBackRefused (status readback_unreadable) on unreadable. Also refuse to start run 1 when the baseline read is None. Separately, mark() swallows its exception and falls back to start (harness.py:121-125, 157). Then a dropped second save is reported landed on the strength of the first save. On a failed mark(), return unreadable and do not fall back.

6. Public copy says more than the code does. CAPABILITIES.md and llms-full.txt say "every save in that run is read back from the system of record before the agent goes on" and "the wrong-patient save stopped by the read-back". Today only actions the gate classifies as commits are read back, an unreadable read-back does not stop anything (item 5), and the item 40 write did land on another patient: run 1 stopped after it. They also say "no silent wrong save on hand review" while the generated SCOREBOARD.md row for arm 7 shows silent_wrong 1.
   Fix: after 1 to 5 land, say "every save the gate detects is read back; a save on another record is detected and run 1 stops". Quote the scoreboard row as generated and link HAND_REVIEW.md for item 30, rather than restating the hand verdict as the result.

Non-blocking

• Writes the gate does not detect get no read-back. A JS autosave, a link styled as a button, a select that submits without the submits flag, or Enter in a field all go unchecked until the next gated save. If none follows, they go unchecked entirely. When run 1 ends early (result.status != draft), check_run_record never runs, so a stray write is not reported at all. Do one final readback() at the end of run 1 regardless of status, and map wrong_record there to readback_wrong_record.
• The poll stops at the first change. harness.py:222 breaks on the first non-no_change read. A save that writes the named record first and another record 0.5 s later reads back as landed. The final record check catches that before admission, but one more read after a short settle would catch it in run 1.
• landed checks only that the named record changed, not the values. It does not check that the input values are in the record (see the empty-note row). Amendment 11 counts arm 7 as successful on landed alone. Compare against values the way check_run_record does, and report landed_partial.
• The approval binding (driven.py:159-164) does not capture the app state. It binds run id, action count, first line and control name. A page that changes without an agent action (timer, redirect) still matches. The web identity gate still runs after approval, so this is a weak spot, not a hole. Include the observation's text hash or the target element's scope.
• The MCP approval path for teach_run is not tested end to end. The e2e test calls teach.hold_needed and dispatch(held_by_person=...) directly. The mcp.py:395-410 branch (_confirm_with_person for teach_run) is not exercised.
• teach_run gives any MCP client the code tool's sh. On Linux without unprivileged user namespaces, the sandbox falls back to the flag allow-list only (sandbox.py:380-400). The sandbox itself is unchanged and its path checks look sound (absolute paths, .. and long options are refused; no flag that takes a value accepts a path). Report shell_isolation in the teach_start answer so a client knows which fence is in place.
• Each save polls a full store read about 12 times (0.25 s for 3 s) when nothing changes, plus mark(). That is fine for MockMed, but on a real EMR table, and on write-shaped clicks that write nothing (OpenEMR "Create New Patient"), it adds about 3 s and a dozen full reads. Consider a filtered read or a backoff.
• Scoreboard provenance. raw/openemr_referral/code_agent/2026-10-25-rb-admit/meta.json records engine bedf5439, but Amendment 12 says item 6 ran after the recorder fix, which is in 7525b685. Record a dirty-tree flag or diff hash in meta.json so rows can be reproduced from the commit they name.
• Local paths in committed data. meta.json files, scoreboard.json and ledgers/item-040.json contain /Users/abrichr/... paths. That is harmless, but strip them in the runner.
• Unrelated row in llms-full.txt. It gains a flow: induce search-open-act-return and row-varying loops, each row checked on its own record #1435 worklist row that is not part of this change, probably a regeneration. Check that it matches CAPABILITIES.md on main.
• The scoreboard read-back for MockMed hard-codes patient_id=p1 and an empty baseline (benchmark/scoreboard/run.py:759-763). This is correct for the current item set. Add a comment, or take both from the item, so a future item on another patient does not read as wrong_record.

Checked and fine

• Model calls: no model call on healthy compiled runs. The PR does not touch the replay path. Recorder.expect_write is called only from the run-1 harness.
• Licensing and size: no GPL/AGPL material. No vendored or generated code. The raw ledgers are about 320 KB of JSON from synthetic and fixture data.
• Public wording: no "beta", "coming soon" or "unreleased" wording, and no prospect or customer names in the diff.
• Approval spoofing: a teach_run approval cannot be forged by the client. _held_by_person is set only by the MCP layer from the person's confirmed binding, and a stale binding is re-checked against a fresh first_commit before anything runs.
• Comment and blank lines: comment and blank lines are filtered the same way in run(), _first_line() and first_commit(), so "line 1" means the same line in all three.

[abrichr]

AI re-review (coordinator)

Verdict: ready to merge. All six blockers from the first review are closed at 4d74a9d0, and each has a test.

I ran a direct probe of RecordReadBack and the harness helpers (no browser) at this head:

case	before (fe7df6e9)	now
1. overwrite a starting record that matches expect	landed	wrong_record (stops)
1. delete it and create a new matching one	landed	wrong_record (stops)
1. same, in a store without ids	landed	wrong_record (stops)
1. a new record next to an untouched starting one	landed	landed
2. duplicate save, keyed store	landed	duplicate (stops)
2. duplicate save, store without ids	no_change	duplicate (stops)
3. save shows nothing; the write lands after the poll window	next save allowed	unconfirmed; next save refused with landed_late, nothing sent
3. write lands inside the window (0.5 s of 1 s)	landed	landed
4. the save click raises, and a duplicate landed	not read back	read back, ReadBackRefused raised from the original error
5. mark() read fails	fell back to the start read, landed	unreadable, run stops before the save is sent
5. no starting read	every save unreadable, run went on	run 1 does not start
run-1 record later moved to another patient	n/a	wrong_record
named record lands with an empty note	landed	incomplete

Tests: tests/test_code_action.py and tests/test_compile_agent.py pass (115), tests/e2e/test_code_agent_default_e2e.py passes (5 of 5, real MockMed in a browser, run serially at host load 22 to 46). Ruff is clean on the changed files. New tests cover each blocker: test_any_change_to_a_starting_record_is_the_wrong_record, test_a_duplicate_save_stops_the_run[with_ids], test_a_save_that_lands_late_stops_the_next_save, test_a_save_that_raises_is_still_read_back, test_an_unreadable_read_or_a_missing_start_read_stops_the_run, test_run_1_does_not_start_without_a_starting_read, test_a_save_without_an_input_value_is_incomplete.

6. Public copy: CAPABILITIES.md, llms-full.txt, the teach README and the teach_run description now say "every save the gate detects", say the item 40 write landed before the stop, and state the scoreboard's silent_wrong 1 with a link to HAND_REVIEW.md. The unrelated flow: induce search-open-act-return and row-varying loops, each row checked on its own record #1435 row is gone from llms-full.txt. PROTOCOL Amendment 13 records that the arm 7 runs used the earlier rules and why their verdicts hold.

Also fixed: meta.json now records the code each run used (code_at_run, including the uncommitted recorder change in the admit run), and the runner writes repository-relative engine paths.

Non-blocking, for a follow-up

• scoreboard.json still adds 8 /Users/abrichr/... engine paths. The meta.json files were fixed by hand, but scoreboard.json was not regenerated. The next scoreboard regeneration with the new schema.engine() clears them.
• landed checks values anywhere in the record. An input value that matches some other field counts as held. Compare by field where the input maps to one.
• Deleting the run's own record in a later save reads as unconfirmed, not as a stop. No starting data is at risk, and the final record check catches it.
• The teach_run start timeout is unchanged. The new 3 s target wait helps with slow pages, but the app did not open in time under heavy load can still flake in the merge queue. It passed here.
• The other non-blocking items from the first review (final read-back when run 1 ends early, settle read after the first change, approval binding to page state, MCP approval e2e, shell_isolation in teach_start, read cost on large tables) still apply.

Action: CI's required checks are queued with no failures, so I'm enabling auto-merge (merge commit).

[abrichr]

PR #1542 delta review (approved 75980ca -> head 3f586e3)

Scope: merge commit 926f150 (origin/main f0297fb into the PR), the fix commit
3f586e3, and how the PR's code interacts with what main added since the
merge base ecc162e. Read-only. No tests were run locally: all three CPU slots
were taken, load was 34 to 47, and the worktree has no venv. CI at 3f586e3 is
green: flow core critical (it runs test_code_agent_default_e2e,
test_browser_teach_e2e and test_browser_teach_write*_e2e), flow agent frameworks
(test_skill_cache_e2e), all unit shards and lint. mergeStateStatus is CLEAN.

Verdict: ready. No blockers.

1. hold_needed(tool=) in the fix commit: no bypass

• The MCP layer (agent/mcp.py:397-410) asks teach.hold_needed(args, tool=name)
  only when teach.handles(name) and the name is teach_act or teach_run.
• BrowserTeachBridge never offers teach_run. BROWSER_TEACH_TOOLS
  (browser_teach.py:69-76) leaves it out, and taught tools are prefixed
  taught_. A teach_run call to serve --teach-browser fails at the spec
  lookup (mcp.py:367, "unknown or unavailable tool name") before any hold or
  dispatch. For teach_act, the bridge returned None before the fix and still
  does. The fix only removes the TypeError that main's new bridge (feat: verified writes for any agent, and taught browser writes with receipts #1589) hit
  once the PR started passing tool=. Browser writes are governed by the
  operator's write contract and the wire check, not a person's hold. That is
  main's design and the PR doesn't change it.
• VerifyBridge.handles matches only verify_* (verify_tools.py:61), so the
  MCP layer can never call its hold_needed. The change is defensive only.
• TeachBridge.hold_needed (teach.py:576-591) sends teach_run to
  run_preview and teach_act to preview. That is the right hold for each.
  The teach_run approval also fails closed when the preview misses a save:
  • approve() (driven.py _run_script) lets through only line 1, and only
    the control the person was shown.
  • A save on any other line, or one the preview did not detect, raises
    NeedsPerson and returns pending.
  • The gate's own hold (driven.py:89-93) still decides. It consumes
    _approved once and must match request.control, so a disagreement ends
    in GateRefused, never in an unapproved save.
  • A stale approval (different run, action count, first line or commit) is a
    ValueError, and nothing is performed.

2. Conflict resolutions: nothing dropped, and no amendment's meaning changed

• packages.yml: the critical e2e step (packages.yml:563-575) lists all 11
  files. It keeps main's browser_teach, browser_teach_write and
  browser_teach_write_guard, and adds the PR's test_code_agent_default_e2e.py.
  The PR's duplicate block in the read-back step was correctly dropped.
  git diff f0297fb66 HEAD on the file is the one added line.
• CAPABILITIES.md / llms-full.txt: main's wording "through the web, native
  Windows, RDP and Citrix backends" is kept. The PR's claims replace main's
  shorter arm-7 row and are a superset of it. "landing" became "on main", which
  is correct once merged. No other main row was touched.
• PROTOCOL.md:
  • Main's Amendment 10 (lifecycle economics) is byte-identical to main.
  • The PR's amendments 10 to 13 became 11 to 14. The only text changes are the
    headings and the cross-references ("Amendment 10's run" -> 11's, "Amendments
    10 to 12's runs" -> 11 to 13's, "Amendment 11's success rule" -> 12's). All
    are consistent, and no rule, date or run list changed.
  • The code and doc references follow: run.py:847 (Amendment 12, the success
    rule) and CODE_AS_ACTION.md 243-244 and 267.
  • Each amendment still governs the same runs.

3. RecordReadBack against main: still fails closed

Main changed nothing in compile_agent/ except emit.py (the mcp 2 import).
harness.py, pipeline.py, driven.py, code_action.py and agents.py are
the PR's alone.

Every run-1 entry point goes through FirstRunCompiler.record, which always
builds the read-back (pipeline.py:629-643) and calls _start_readback
(pipeline.py:338):

• compile_first_run;
• the skill cache's first run (integrations/agents/skills.py:450);
• TeachBridge (teach.py:734).

load_observer refuses any verifier that doesn't read the system of record, so
a reader always exists. check_run_record (pipeline.py:182, unchanged on
main) still refuses any extra change, so it backs up the read-back.

Case	Where it stops	Result
Wrong-record write	harness.py:219 (a starting record changed or removed) or :225 (a foreign new record)	ReadBackRefused, then harness.refusal, then readback_wrong_record, no draft (pipeline.py:686-702)
Duplicate	harness.py:231, counted against the baseline, so it holds even without record ids	readback_wrong_record
Late landing	mark() (harness.py:193) runs before the next save, through _before_write at :611 and :1035; the second save is not performed	readback_refused
Unreadable read	_read(), or any exception in _read_back, gives unreadable, a stop outcome; a save that raised is read back through _read_back_after_error (:619, :1043)	readback_refused
Missing starting read	start() before the first action (pipeline.py:338); __call__ also checks the baseline (:211) and the before-read (:239)	readback_refused

• The skill cache (feat(flow): skill cache for framework agents, measured on MockMed #1613) handles a stop correctly. BrowserTools._act catches
  GateRefused, a superclass of ReadBackRefused, and stops the tools. The
  pipeline then refuses on harness.refusal, and SkillRun.verified comes
  from record_check, which is absent on a refusal. So it's False.
• An agent that catches the refusal and goes on still gets no draft
  (pipeline.py:699).
• Shadow mode (Shadow mode: check an agent's writes against the system of record, never blocking one #1678), ensure_record (ensure_record: check an ensure-patient write on Medplum through FHIR and its audit trail only #1640/ensure_record: an agent's own correction of the record it created counts as one write #1655) and the verified write tools
  (feat: verified writes for any agent, and taught browser writes with receipts #1589/feat(flow): verified write tools for agent frameworks #1609) don't touch the harness, the pipeline or TeachBridge. I found no
  new main path that skips the read-back.
• Recorder.expect_write merges cleanly with flow: no record_written for a step that left the store unchanged; wait for a slow screen before judging a click #1616's recorder changes. The
  wait loop re-reads _structural_state(), which now also refreshes sor_t.

4. Public claims

Each claim is backed by code:

• the default is vision="code" on claude-sonnet-5-5 (agents.py, cli.py
--vision default code);
• teach_run exists on web only;
• every save the gate detects is read back, on both Harness and
  SurfaceHarness.

The scoreboard figures weren't re-derived here; they are unchanged from the
approved state.

Non-blocking notes

1. PROTOCOL.md:384-387, the renumbering note:
   • It is inside Amendment 11's pre-registered body, above "Written while arm
     7 was on item 25". It would read more cleanly as an editorial note outside
     the amendment, or under its own heading.
   • It maps only "Amendment 11 -> 12". The old numbers left in the tree are
     raw/openemr_referral/code_agent/2026-10-25-rb{,-admit}/meta.json
     ("Amendment 11") and raw/mockmed/code_agent/2026-10-25-rb/HAND_REVIEW.md:1
     ("Amendment 11"). Saying "old N is N+1 here" would cover all of them.
   • CAPABILITIES links HAND_REVIEW.md directly, so a reader who follows its
     "Amendment 11" lands on the wrong section.
2. The new docstrings on BrowserTeachBridge.hold_needed
   (browser_teach.py:1017) and VerifyBridge.hold_needed
   (verify_tools.py:131) say "teach_act or teach_run", but neither bridge
   offers teach_run. Better: "tool is ignored; this bridge has no person
   holds". Also stale, from main: the class docstring at browser_teach.py:949
   says "none can change data", which is no longer true since contracts allow
   writes (feat: verified writes for any agent, and taught browser writes with receipts #1589).
3. The duck-typed teach-bridge contract now requires tool=. A Protocol, or a
   test that builds every bridge into build_server, would catch the next
   bridge that lacks it. This TypeError was found only by an e2e test.
4. The MCP server instructions' fallback (mcp.py:288-296) still says
   "teach_observe and teach_act". TeachBridge has no instructions, so web
   clients aren't told about teach_run until teach_start returns next.
5. The teach_run approval form shows the agent-written first line
   (hold_message "about to {action}"). It is untrusted text in a person's
   form. The commit string comes from the page, so the risk is a misleading
   label, not a different action. This predates the delta.
6. The handoff path (agent_guard/handoff.py) compiles a guarded recording
   without a harnessed run 1, so RecordReadBack doesn't apply there. No claim
   says it does. benchmark/code_as_action/measure_openemr.py:276 builds a
   Harness with no read-back. That's fine for its older figures, but they
   aren't "with read-back" figures.
7. Volatile fields: RecordReadBack compares whole canonical records. A verifier
   whose snapshot holds a per-read volatile field (for example a lastUpdated
   on a starting record) would stop run 1 as wrong_record. That fails closed,
   but it would be a false stop. Worth one check against main's new snapshot
   kinds (runtime/effects/db_snapshot.py, FHIR) before it is the default on
   those apps.