Skip to content

flow: run a form's own script before an API save instead of refusing it - #1540

Merged
abrichr merged 29 commits into
mainfrom
claude/api-step-page-script
Nov 3, 2026
Merged

abrichr merged 29 commits into
mainfrom
claude/api-step-page-script

Conversation

@abrichr

@abrichr abrichr commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

What this changes

Until now the compiler refused an API step for a form whose page runs script on the save: an onsubmit validator, on* handlers on the fields it sends, a page listener for submit, or a validate.js rule kind it can't evaluate in Python (OpenEMR's pastDate). That refusal is one of the two that keep the OpenEMR referral's save on the GUI.

Now the API step runs that script instead of refusing the form:

  • Compile records the script in the proposal (web_form.client_scripts). The admission digest covers it, and api show lists it.
  • Run. Before each save, runtime/actuators/form_eval.py opens the form page in a scratch copy of the live session. That is a separate browser context with the session's cookies, so the run's own tabs are untouched.
    • The route blocks every request that isn't a GET, and every script-sent request once the page has loaded. Nothing the page does there can write.
    • The run's typed and picked values go in with the same input, change and blur events a browser fires.
    • The form is then submitted through its own handlers. A final listener records whether they let the submit through, then stops it. HTMLFormElement.prototype.submit is counted and never navigates.
    • Declared validate.js constraints run too.
  • The save is sent only when the page let the submit through, raised no error, showed no dialog, and tried to send nothing. The browser must also be about to send exactly the body the API step built (session values aside). Anything else is a reason, and the GUI steps run.
  • A live form whose script differs from the recorded script doesn't run the API step.

Nothing here loosens what was already checked. Native constraints and the evaluable validate.js rules are still checked in Python first; the executed check is added on top.

Evidence

  • tests/test_api_step_page_script.py: 17 tests with a real headless Chromium against a local server. Cases:
    • an accepted value;
    • a custom rule refusing a value;
    • a handler that rewrites a value, where the actuator refuses the mismatch;
    • script that tries to POST, which is blocked and unevaluable;
    • a programmatic submit();
    • a load error, an alert, stopPropagation, native required, validate.js constraints;
    • an unknown option, a missing form;
    • checks that the run's tab is untouched and that the server saw no write.
  • The existing API-step tests pass, and a local sweep of the API, replayer, authorization, qualification, durable, approval, policy and compatibility tests: 2017 passed, 3 skipped, 3 xfailed, 0 failed. The test that asserted the old refusal now asserts that the script is recorded and must be run.
  • Offline, on the referral recording the bench(flow): referral API step under each_item approval + durable, 20 runs per arm #1374 measurement used: the script refusal is gone. The proposal still stops on the second reason, POST /library/ajax/execute_background_services.php from the tab shell. The stacked PR handles that.

Not yet measured on a live app. The OpenEMR measurement runs once the fixture lease comes round, on the stacked PR, which is what lets the referral through.

Limits (in API_TIER.md)

Script that validates only in a click handler of the submit button, or asynchronously after the submit event, isn't seen. Qualification on both paths is what shows the paths agree. A form whose script needs the server while it validates stays on the GUI. Each save with script costs one extra load of the form page.

🤖 Generated with Claude Code

Review fixes (second push)

  • Body at submit time. The body compared is the one the page submits, captured when it submits: by a last submit listener, by the form's own submit() (instrumented), or as the browser's POST, which the route blocks. A rewrite in onsubmit is now caught, and any difference keeps the save on the GUI.
  • Click handlers. The proposal records the control the person clicked in the form (submitter, a button or a submit/button/image input) and the requests that click made (submitter_requests). The scratch run clicks that control, and the page must then submit or make one of those requests. On the referral, that is #create and GET /interface/new/new_search_popup.php. With no recorded control, a form whose buttons have click handlers is refused.
  • Blocking. The scratch context now has no service workers and lets nothing from other origins through. Once the page has loaded, it blocks every request. It never opens a tab in the run's own browser context.
  • Fingerprint. client_scripts holds every script URL, whether the page has inline script, and each handler (form, sent fields, buttons) with a digest of its source. So the page's script runs whenever the page has any, and a changed script or handler keeps the API step off.
  • Equivalence claim. API_TIER.md no longer claims the two paths are proven to agree.

Targeted tests after merging main: 244 passed, 1 skipped.

abrichr and others added 5 commits October 24, 2026 23:39
A form whose fields have handlers, whose onsubmit validates, or whose
validate.js rules use a custom kind (OpenEMR's pastDate) kept its save on
the GUI. The compiler now records that script in the proposal
(client_scripts), and before each save the runtime runs it: the form page
opens in a scratch copy of the live session with every non-GET request and
every script-sent request after load blocked, the run's typed and picked
values go in with input, change and blur events, and the form is submitted
through its own handlers with a final listener that records the verdict and
stops the submit. The save is sent only when the page let the submit
through, raised no error, showed no dialog, tried to send nothing, and
would send exactly the body the API step built. Otherwise the GUI steps run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any other write sent while the replaced steps ran refused the API step.
Each is now declared in the proposal (web_form.writes), in recorded order:

- ambient: a script request from another document than the form, loaded
  before the form opened, to an endpoint the form page doesn't name (for
  example OpenEMR's tab shell posting execute_background_services.php on a
  timer). No replaced step caused it, so the API step doesn't send it.
- step: a write a replaced step caused, rebuilt like the save and sent in
  its recorded place before it. It is read back by the record check of the
  step it was sent during (admission refuses a step write whose step has
  none; the binding's record check is the save's plus those), and a failure
  or an answer outside its recorded status stops the run with nothing after
  it sent and nothing retried. The approved digest covers every request.

Anything else still refuses the proposal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Interleaved browser and API arms on the API-tier fixture, the eight
referrals plus a future date of birth (the form's pastDate rule must refuse
it) and a lower-case first name, a per-item timeout, the test setup's own
SQL read-back, and the whole saved patient_data row per run so the two
arms' saves can be compared column by column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@abrichr abrichr left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Coordinator safety review (AI-assisted). Changes requested.

  1. form_eval.py:77 vs :90. FormData is read before requestSubmit(), so rewrites made inside onsubmit, or just before a programmatic submit(), are never compared. Capture FormData inside the submit listener and the instrumented HTMLFormElement.prototype.submit, compare that body, and document that any difference is refused.
  2. web_form.py:89 and :703. An onclick validator on the submit button and an external .js listener go undetected, so form_eval is skipped and the save is sent. That's the case where the GUI refuses and the API saves. Run form_eval whenever the page has any script, and refuse (or click the real submit control) when a button in the form has a handler.
  3. form_eval.py:170-174. Cross-origin script and style loads go through, and so do same-origin page loads after load; the scratch context also shares session cookies. Block non-origin requests and all page loads after load.

Smaller:

  • Hash the handler source and script URLs into client_scripts.
  • The GUI-and-API equivalence proof the docs claim doesn't exist yet. Add it, or reword.

abrichr and others added 10 commits October 25, 2026 01:26
- Compare the body the page submits, taken when it submits: a last submit
  listener, the form's own submit(), or the browser's POST the route blocks.
- Click the control the person clicked (recorded as submitter), and require
  the page to submit or make a request that click made in the recording;
  with no recorded control, refuse a form whose buttons have click handlers.
- Run the page's script whenever the page has any: client_scripts now holds
  every script URL, inline script, and each handler with a digest of its
  source (form, sent fields, buttons).
- In the scratch context, block every other origin and every request once
  the page has loaded; service workers are blocked; no scratch tab in the
  live context.
- API_TIER.md no longer claims the two paths are proven to agree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he click's

Events are stamped once the screen settles, so a click's requests fall
between the previous and the next event, and stop at the save. Only a
button or a submit, button or image input counts as the control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Any other write sent while the replaced steps ran refused the API step.
Each is now declared in the proposal (web_form.writes), in recorded order:

- ambient: a script request from another document than the form, loaded
  before the form opened, to an endpoint the form page doesn't name (for
  example OpenEMR's tab shell posting execute_background_services.php on a
  timer). No replaced step caused it, so the API step doesn't send it.
- step: a write a replaced step caused, rebuilt like the save and sent in
  its recorded place before it. It is read back by the record check of the
  step it was sent during (admission refuses a step write whose step has
  none; the binding's record check is the save's plus those), and a failure
  or an answer outside its recorded status stops the run with nothing after
  it sent and nothing retried. The approved digest covers every request.

Anything else still refuses the proposal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Interleaved browser and API arms on the API-tier fixture, the eight
referrals plus a future date of birth (the form's pastDate rule must refuse
it) and a lower-case first name, a per-item timeout, the test setup's own
SQL read-back, and the whole saved patient_data row per run so the two
arms' saves can be compared column by column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A durable first leg now persists, before the span's first request leaves,
the folded steps' checkpoints and a delivery-uncertain pause at the save.
A crash anywhere in the sequence of writes then resumes at the save, which
is only reconciled, never sent again. The save's own outcome replaces that
pause, or, once its checkpoint is persisted, the pause is consumed (only if
it is still exactly the one written). A resumed leg keeps the continuation
guard's delivery_started phase as its write-ahead. api admit lists the
requests the API step leaves to the page before the person types admit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before each save, the page's script runs on the run's values in the scratch
page; each declared step write must be one it sent there (blocked), with
the same body, or the GUI steps run. A span with step writes and no page
script to check them against stays on the GUI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

AI re-review (coordinator). Changes requested: one blocker, plus three ways the API save can send what the GUI wouldn't.

The three original findings are addressed in the code paths they named: the body is captured at submit, the recorded control is clicked, cross-origin and post-load requests are blocked, and client_scripts carries digests. I probed head 5bc618d49 with real headless Chromium against the test suite's _App server. Each case below is a small page fed to PlaywrightSessionTransport.evaluate_form.

Blocker: the scratch run can make a real write. form_eval.py:256 routes only the scratch page (scratch.route). A popup in the same context isn't routed. Take a form with target="_blank" whose script calls e.stopPropagation() in a form-level submit listener: the window-level last listener never runs, the browser's own submission opens a new page, and that page POSTs /save.php to the server with the session's cookies. Reproduced: app.writes == ['/save.php']. The verdict is unevaluable, so the GUI then saves a second time. A submitter with formtarget="_blank", or a link with target=_blank, has the same effect.
Fix: context.route("**/*", route) instead of scratch.route, plus context.on("page", ...) that marks the result unevaluable and closes any new page. Add a test with target="_blank" and stopPropagation.

WebSocket bypasses the post-load block. Playwright routes don't intercept WebSockets. A change listener that runs new WebSocket('ws://'+location.host+'/ws') reached the server after load with the session cookie, and the verdict was valid. A WebSocket to another origin would also get through.
Fix: context.route_web_socket("**", lambda ws: ws.close()) (Playwright 1.48 or later), or stub WebSocket, EventSource and RTCPeerConnection in _INSTRUMENT_JS and record any use as unevaluable. (sendBeacon is caught and blocked: verified.)

The GUI and the API save can differ:

  1. Key handlers never run (form_eval.py:90-94). Values go in through el.value = plus input, change and blur, so keydown, keypress, keyup and beforeinput handlers are skipped. With <input name="lname" onkeyup="this.value=this.value.toUpperCase()"> the verdict is valid with lname=Marsh, while the GUI would send MARSH. OpenEMR date fields use onkeyup="datekeyup(...)", so this affects the target app.
    Fix: type with locator.press_sequentially() (trusted key events), or refuse when a sent field has a key*/beforeinput attribute or listener. Extend _INSTRUMENT_JS's listener tracking to those event types.
  2. The script sees untrusted events (form_eval.py:91-93, :120). el.click() and dispatchEvent produce isTrusted === false. A handler that runs if (!event.isTrusted) { this.form.submit(); return; } before its validation gets valid with dob=2999-01-01, a value its own check refuses.
    Fix: scratch.locator(submitter).click() and keyboard typing through Playwright, which dispatch trusted input.
  3. A recorded request counts before its response decides (form_eval.py:368-370). The via=recorded path accepts the page as "went on as recorded" when it starts the recorded request (the duplicate search). In the GUI the response decides. With fetch('/dup.php').then(t => t.includes('MATCH') ? alert('duplicate') : f.submit()) and a server that returns a match, the GUI shows the duplicate alert and saves nothing. The scratch verdict is valid via GET /dup.php, so the API save creates the duplicate.
    Fix: let the exact recorded same-origin GET through in the scratch run, then require the page to actually submit (reached, programmatic, or the blocked save POST) within the settle time. Treat "made the recorded request" alone as unevaluable.

Smaller: only saves[0] is compared (:354). A page that POSTs the save twice should be unevaluable.

Not merging; the PR stays a draft. #1544 stacks on this one and inherits these issues.

abrichr and others added 4 commits October 25, 2026 02:24
Second review of the scratch run:
- route every page of the scratch context (a popup is unevaluable and is
  closed), close every WebSocket, stub EventSource, RTCPeerConnection,
  sendBeacon and window.open;
- type the values with real key events so key handlers run (fields not on
  screen get the same events from script), and click with real, trusted
  clicks;
- a recorded read now completes, the recorded confirm control is clicked
  in whichever frame has it, and the page must then submit; a read alone no
  longer counts;
- a page that submits more than once is unevaluable; after load, an image
  request that isn't a static file counts as the page sending.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h run

The scratch run answers a declared step write with an empty 200 (it never
reaches the app) and returns its body; each request the page sent matches
at most one step write, and an extra one refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…losed

A field that isn't on screen when the form opens (on a tab the person
opened first) made the scratch run use script events. It now makes the run
unevaluable, so the save stays on the GUI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

AI re-review 2 (coordinator), head 10f2f0978. Changes requested: one blocker remains.

The earlier five probes are fixed (real headless Chromium, the suite's _App server):

  • target=_blank plus stopPropagation: the new page is closed and the verdict is unevaluable. No write reached the server.
  • WebSocket: blocked, unevaluable.
  • An onkeyup uppercase handler runs, and the compared body has MARSH.
  • An isTrusted-gated validator: the trusted click runs it, so the verdict is invalid.
  • Duplicate search with a match: the recorded GET completes, the page shows its alert, so the verdict is invalid.

New checks that hold:

  • <link rel=prefetch>, a CSS url() background to a .php path, and fetch(..., {keepalive: true}) are all blocked and make the run unevaluable.
  • With no recorded control, requestSubmit() runs the onsubmit validator, so the verdict is invalid.
  • A WebSocket opened from a fresh iframe's own constructor (which gets past the init-script stub) never reached the server, because route_web_socket caught it.

Blocker: after load, any same-origin script or stylesheet GET goes through (form_eval.py:222-230). The allowance is by resource type, not by URL. A change handler that appends <link rel="stylesheet" href="/act.php?delete=1"> or <script src="/act2.php?delete=1"> got both GETs to the server with the session cookie, and the verdict was valid. That's a GET with side effects running from the scratch run. It runs again on every scratch run, including the ones that end invalid or unevaluable and then run the GUI. It also contradicts the docstring's "nothing the page does can write".
Fix: record the script and stylesheet URLs requested during load. After load, continue only those exact URLs (or the recorded reads), and log anything else in late and abort it. A page that lazy-loads new code then falls back to the GUI. Add a test with a dynamically added stylesheet or script to a .php URL.

Non-blocking, as the coordinator decided:

  • Fields hidden when the form opens (form_eval.py:412-415) still proceed with script-dispatched, untrusted key events. With style="display:none" onkeyup="if (event.isTrusted) ..." the verdict was valid with the untransformed value. Those fields should make the run unevaluable so the step falls back to the GUI. That isn't required for merge, but the docs should say this is how it currently works.
  • A form inside an iframe of the page: that frame's form.submit() is stubbed by the init script, but its calls are recorded in the frame's own state, which _COLLECT_JS never reads. The verdict was valid while the GUI would also POST the iframe's form. Nothing leaked. A recording that saw that write is caught by other_writes_seen, but collecting programmatic from every frame would make this airtight.

Not marked ready, and no auto-merge.

abrichr and others added 2 commits October 25, 2026 02:39
… get through

Round 3: a script or stylesheet passed by resource type alone, so a handler
could add <link href=/act.php?...> and reach the app with the session. Now
only the exact script and stylesheet URLs the page loaded, the recorded
reads, and the static .js/.css files of a recorded read's own frame get
through; anything else is blocked and makes the run unevaluable. Submits
and blocked channels are collected from every frame, not only the form's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-sequence

# Conflicts:
#	packages/flow/openadapt_flow/runtime/actuators/form_eval.py
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

AI re-review 3 (coordinator), head f02890763. No blocking issue. Marking ready and enabling auto-merge.

Probes, run in real headless Chromium against the suite's _App server:

  • After load, a handler that adds <link rel=stylesheet href="/act.php?delete=1">, <script src="/act2.php?delete=1"> or <script src="/x.js"> from the form's frame is unevaluable. No GET reached the server.
  • form.submit() from a form inside an iframe is now collected ("a form submit in …/inner.php") and is unevaluable. An iframe that does nothing still gives valid.
  • A field hidden when the form opens is now unevaluable, so the step falls back to the GUI, as decided.
  • All the earlier probes stay fixed: the popup, WebSocket, key handlers, trusted clicks, the duplicate-search response, prefetch, CSS url() and keepalive.

The static .js/.css exception is accepted. It's narrow:

  • a same-origin GET of type script or stylesheet;
  • no query string, with a path ending in .js or .css;
  • requested from a frame whose own URL is a recorded read (the confirm dialog).

The GUI loads that same dialog and its assets. For this to have a side effect, the app would need to route a query-less /x.js to a handler that writes, which isn't a realistic pattern for OpenEMR or typical PHP front controllers (their static paths are served by the web server). Requiring the exact URL from the GUI recording would make the step fragile (cache-busting renames) for little gain. If a target app ever routes asset paths to code, tighten this to recorded URLs.

Targeted tests pass on the #1544 head, which includes this branch: 94 passed across the page-script, web-form, durable and write-sequence suites. That's about 850 added non-test code lines, under the 1000 limit.

@abrichr
abrichr marked this pull request as ready for review October 25, 2026 06:45
@abrichr
abrichr enabled auto-merge October 25, 2026 06:45
flow: declare an API step's other writes as an ordered sequence
@abrichr
abrichr disabled auto-merge October 25, 2026 06:45
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

AI re-review (coordinator), follow-up. Arming auto-merge on #1544 merged it straight into its base branch claude/api-step-page-script, which is this PR's head (merge commit 4a16aaa71). That branch has no protection, so the merge didn't wait for checks. This PR now carries both changes: 2988 additions, about 1770 of them non-test code, which is over the 1000-line auto-merge limit. I've turned auto-merge off on this PR. Both parts were reviewed with no blocking issue (see the re-review 3 comments here and on #1544). Re-arm it, or merge once checks pass, after a person decides.

@abrichr
abrichr enabled auto-merge October 25, 2026 06:46
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

Coordinator: #1540 now contains #1544 (merged into its branch). Both parts passed three rounds of adversarial review with no blocker left (comments above). This is the deliberate merge decision for the combined change after full review; auto-merge re-armed so it lands when the required checks pass.

@abrichr
abrichr disabled auto-merge November 3, 2026 17:53
form_eval: the WebSocket route handler was a lambda returning a tuple of
None values; make it a function. api_proposal: narrow the selector value
and the type= match before use. No behavior change.
A guarded create-form save leaves its wire check armed after the click,
and the next step reads it. An admitted API span skipped that read, and
its requests go through context.request, which context routes never
intercept, so its save could go out after the check had aborted a late
submit from the earlier save's page.

_run_web_form_span now reads the check first (without disarming it) and,
on a refusal, returns a safety halt on the span's first step with nothing
sent. Never None: None would run the GUI steps instead. The end-of-run
message now says 'after an earlier save'.

Test: a real browser and server; the save's page re-submits after the
settle window, then an admitted API span on the same page halts at its
first step and no API POST reaches the server.
viewport goes to new_context (new_page doesn't take it). The page's late
submit is triggered by the test after the save's run ends, not by a
timer, so CI timing can't move it. The halt must be the API span's own
(actuation api, no GUI fallback).
…tops its send

Without an effect verifier the API tier halts before the actuator, so the
test passed without the fix. With RestRecordVerifier (as the API span
tests use), the test fails on the pre-fix replayer: the API POST reaches
the server.
Conflict in api_proposal.py: main (#1617) added the carried-guard check
before the other-writes check, which this branch replaced with
_span_writes. Kept both: the carried-guard check, then _span_writes.
@abrichr

abrichr commented Nov 3, 2026

Copy link
Copy Markdown
Member Author

PR #1540 delta review (approved state 4a16aaa -> head 6dc364e)

Scope: commits 541efc6 (merge main), 50dd665 (lint), 6dc364e (merge main),
and how the PR's API span interacts with code main added since approval
(#1581 wire check, #1678 shadow mode). Read-only; no tests run.

Verdict: blocker (one, small fix)

Blocker 1: an API span skips the earlier save's wire-check read, so its save can go out after a refusal

All paths are in packages/flow/openadapt_flow/.

How main (#1581) works:

  • A guarded create-form GUI save arms the wire check in form_identity_at /
    _arm_wire_check (backends/playwright_backend.py:2543, :2598). The check
    is context.route("**/*", handler) on the live context (:2672) and stays
    armed after the click.
  • _act watches it for _FORM_GUARD_SETTLE_S (2 s). After that, it is read
    and disarmed only at (a) the next non-guarded step's _act
    (runtime/replayer.py:13243-13249), (b) the next guarded save's
    form_identity_at (playwright_backend.py:2555), or (c) the end of the run
    (_halt_on_late_form_submit, replayer.py:13271, called at :2599).

What the PR's span does:

  • _run_web_form_span (replayer.py:3029) goes straight to
    _run_step(save), then _try_api_tier. _run_step returns before _act
    when _web_form_span_active is set (:8173), so none of (a) to (c) runs
    before the API save.
  • The API save goes through page.context.request
    (runtime/actuators/web_form.py:524, :560, :569). That is Playwright's
    APIRequestContext, which context routes don't intercept, so the armed wire
    check neither sees the save nor blocks it. The scratch run
    (form_eval.py:305) uses its own browser.new_context, which is also
    outside the live check.

Failure scenario:

  1. Step k is a guarded create-form GUI save. Its click passes, and the 2 s
    settle window shows no refusal.
  2. At about t = 2.5 s, a page script fires a second POST (for example a
    duplicate create). The wire check aborts it and records a refusal. Under
    main's rule, the run must stop here.
  3. Step k+1 starts an admitted API span. _web_form_span_at and
    _run_web_form_span never read the guard. The scratch run passes and the
    API save POSTs a second record through context.request.
  4. The refusal comes out only at the next non-guarded GUI step, or at the end
    of the run. If the span is the last step, _halt_on_late_form_submit
    marks the API save as failed with the message "after the save, ...". That
    misattributes the refusal to a write that has already gone out (and was
    checkpointed as ok in a durable run).

The result is a second write sent after the first one was refused, which
breaks the fail-closed rule. Before this merge, every following action read
the guard first. The PR's span is the only actuation path that doesn't, and no
test covers the combination (tests/test_create_form_wire.py has no API span).

Minimal fix (checked against the code)

At the top of _run_web_form_span, before the write-ahead closure is set up
and before _run_step:

backend = self.backend
if getattr(backend, "form_submit_guard_armed", False):
    late = backend.form_submit_guard_refusal()  # read; see note on disarm
    if late is not None:
        first = workflow.steps[start_index]
        return [StepResult(
            step_id=first.id, intent=first.intent, ok=False,
            safety_halt=True, failure_category="safety_halt",
            actuation="api", delivery_attempted=False, surface=surface,
            error=(f"Step '{first.id}' ({first.intent}): after the previous "
                   f"save, {late} — run aborted"),
        )]

How the caller handles this (replayer.py:2515-2590):

  • A one-element list puts the halt on step_index, and span_skip_to is set.
  • already is False and checkpoint_folded is False, so the halt is recorded
    durably, as a GUI halt would be.
  • stop_run breaks the loop.

It must not return None. None means "fall back to the GUI steps", and if the
guard had already been disarmed, those steps would run with no halt. That
would fail open.

Disarm or keep armed: either choice closes the blocker.

  • Keeping it armed (read with form_submit_guard_refusal() and no disarm) is
    stricter. A stray page submit during the span is still aborted and is
    reported by the next step or by the end-of-run check. The span sends nothing
    through the live page's routes, so it doesn't need the guard off. Risk: a
    record check that navigates the live page with a GET to the form's address
    would be aborted. That fails closed, but the run halts after the write.
  • Disarming (cancel_form_submit_guard()) matches main's non-guarded GUI step
    exactly, but lets a page submit during the span go out unchecked.

I'd keep it armed. Add a test: a guarded save whose page script submits late,
followed by an admitted API span. Assert that no API POST reaches the server
and that the run halts at the span's first step.

Point 1: lint commit 50dd665 (no behavior change, confirmed)

  • form_eval.py:310-318: the lambda returned a tuple only so that it could
    run two statements in one expression. The new on_web_socket does the same
    two things in the same order, and nothing reads the return value.
  • api_proposal.py:261: attr is None or value is None. Both regex branches
    set attr and value together, so value is None only when attr is
    None. This is type narrowing only.
  • api_proposal.py:274: kind is not None and ... is the same as
    bool(kind) and .... A re.Match is always truthy, and the result is still
    a bool.

Point 3: other ways the merge could let an API save skip its check

  • Both merges were clean: git show --remerge-diff shows no conflict hunks.
  • _run_web_form_span is called only from the linear loop (:2528).
    _interpret_program doesn't use spans, so programs keep the GUI path and
    its guards.
  • An API span standing in for a guarded save (its own save has
    form_identity) skips form_identity_at by design. The scratch run is its
    check, and a fallback (None) re-runs the GUI steps, guards included. Fine.
  • Shadow mode (Shadow mode: check an agent's writes against the system of record, never blocking one #1678: shadow.py, cli_shadow.py, shadow_report.py)
    doesn't touch the replayer or the web_form actuator. No interaction.
  • Main's other replayer changes since approval (hover jitter, local model-call
    accounting, pixel census, identity_armed_steps counting form_identity)
    don't touch the span's gates. I found no other way for an API save to go
    out without its scratch-run check.

Non-blocking notes

  • After the fix, the end-of-run _halt_on_late_form_submit can still pin a
    refusal raised during a final API span on the API save ("after the save,
    ..."), although the refused request came from the earlier GUI save's page.
    Consider wording it as "after an earlier save".
  • If the guard is kept armed through the span, say so in the
    _run_web_form_span docstring. Its requests skip the live routes because of
    APIRequestContext, so nobody should later "fix" that by routing them.
  • Line 274's new line is about 89 characters long. Ruff passed in the lint
    commit, so it fits the configured limit; noted only in case the formatter
    rewraps it later.

Re-review of the fix: d385ed0 (on 6dc364e)

Verdict: blocker, because the committed test is broken. The runtime fix is correct.

Blocker: the new test crashes before it reaches the case

packages/flow/tests/test_create_form_wire.py in d385ed0 calls
browser.new_context(service_workers="block").new_page(viewport={...}).
BrowserContext.new_page takes no arguments. In this repo's venv its
signature is new_page(self) -> Page, so the call raises TypeError and the
test errors before it gets to the case. The worktree has an uncommitted edit
(not mine; I left it alone) that fixes this by moving viewport= into
new_context(...). Commit that edit, and the blocker is gone.

1. The fix closes the blocker on every path into _run_web_form_span

  • _run_web_form_span has one caller, the linear loop (replayer.py:~2528).
    The guard read is the first thing after router/surface are computed. It
    comes before the folded results, the write-ahead closure,
    _web_form_span_active and _run_step. So the approval view, the scratch
    run, prechecks and the send all come after it.
  • On a refusal it returns a one-element list and never None, so the GUI steps
    don't run.
  • How the caller handles that list: save_result is the halt. already is
    False because len(batch) == 1, and checkpoint_folded is False. The halt
    is recorded at step_index with durable_run, as a GUI halt would be;
    ok=False makes _record_linear_result stop the run, and the loop breaks.
  • Leftover state is safe. The early return skips the write-ahead setup, but
    _web_form_write_ahead is reset to None in the previous span's finally
    and initialised to None (:1195). A stale _web_form_folded_checkpointed
    is read only when len(batch) > 1.
  • A resume leg doesn't go through this function
    (_resume_inside_web_form_span). A resume also runs in a new process with
    an unarmed backend, so nothing is missed there.

2. The check stays armed through the span, and a page submit is still caught

  • The fix uses form_submit_guard_refusal() without disarm. The context
    route stays in place, so a stray page write during the span is still
    aborted at the network.
  • That abort is reported by the next non-guarded step's _act, the next
    guarded save's form_identity_at, or _halt_on_late_form_submit at the
    end of the run. Its message now reads "after an earlier save", which is
    right.
  • Limit: a refusal raised during the span comes out only after the span's
    save has been sent. That is inherent to the design and not a regression.
  • The docstring explains why the span's requests must not be routed.
  • If the span falls back (returns None), the first GUI step's _act reads
    and disarms the check as before.

3. The test exercises the case, with two weaknesses

What it does right:

  • It uses a real Chromium browser and a real server.
  • The page sends a late fetch POST 2.6 s after the submit, past the 2 s
    settle window. The wire check aborts it.
  • An admitted span then runs on the same backend. Its actuator posts through
    page.context.request, the same path the real transport uses.
  • It asserts that the server received only the GUI save, that the actuator
    was never called, and that the run halted at step_000 with
    safety_halt, failure_category="safety_halt", no delivery attempt and
    the "after the previous save" message.
  • Without the fix, the actuator's POST reaches the server, so the test fails.
    It tells the fixed and unfixed code apart.

Weaknesses (non-blocking):

  • (a) Not specific to the span. If the span ever stopped being admitted
    for this workflow, the GUI fallback's step_000 _act would halt with the
    same message shape, on the same step, with no API call. The test would then
    pass without the span ever running. Add
    assert halt.actuation == "api" and halt.api_fallback is None.
  • (b) Timing can fail it under load. The save and the span run as two
    Replayer.run calls on one backend. That is the same state the
    single-run case reads, so the coverage is fine. But if the first run ends
    more than 2.6 s after the click (for example on a busy CI machine), its
    end-of-run check sees the refusal, saved.success is False, and the test
    fails. It fails closed rather than passing falsely, but it may be flaky.
    Firing the late submit from a page hook after the first run returns would
    make it deterministic.

4. Nothing new breaks

  • An unarmed backend, or one without the attribute, is handled by
    getattr(..., False), so nothing changes for it.
  • A backend that reports armed but has no form_submit_guard_refusal would
    raise AttributeError. Main's _act makes the same assumption, and only
    PlaywrightBackend arms the guard, so this is acceptable.
  • Durable path: covered above. The halt is checkpointed at the span's first
    step and no write-ahead pause is created.
  • Approval path: the guard read comes before _run_step, where the approval
    view is built, so nobody is asked to approve a save that would be refused.

Test run

See the result note below.

Result: not run. The run waited about 25 minutes behind
~/oa/scratch/cpu-slots/run.sh. All three slots stayed taken by other
sessions' pytest runs, and the load was 28 to 130. I stopped my queued job
without it starting. Nothing in this re-review comes from a test run; it is
all from reading the code. That the committed test errors is confirmed from
the Playwright signature in the venv, not from running the test. Once the
uncommitted edit is committed, run the two files through the slot runner
before merging.

Summary: the runtime fix closes the blocker on every path. The PR is
ready once the test's new_page(viewport=...) crash is fixed and committed
(the worktree's uncommitted edit does this). I'd also add the
actuation == "api" / api_fallback is None assertion.


Final re-review: head 62fe325

Verdict: ready

Two non-blocking notes, below.

1. The test commits (0fcf7f7, ddadecd)

  • viewport now goes to new_context(...), so setup no longer crashes.
  • The new assertion actuation == "api" and api_fallback is None pins the
    halt to the span. A GUI fallback can no longer make the test pass.
  • window.__sendLate() is called after the save's run returns, so the order
    no longer depends on wall-clock timing.
  • The span now has a RestRecordVerifier. With it, the API tier gets as far
    as the actuator, so the wire-check read at the top of _run_web_form_span
    is the only thing stopping the send.
  • The test now fails without the fix. I put the ddadecd test file onto
    6dc364e in a temporary detached worktree, with PYTHONPATH pointing at
    that tree; the pre-fix _run_web_form_span has no guard read. Run through
    the slot runner with -n 2, the test fails for the right reason
    (test_create_form_wire.py:1182): the server received
    {'field_summary': 'sent by the API step'} after the GUI save. 1 failed,
    94.7 s. I removed the temporary worktree afterwards.
  • At head (62fe325),
    pytest -n 2 tests/test_create_form_wire.py tests/test_api_step_page_script.py
    passed: 79 passed in 248 s.

2. The merge conflict in compiler/api_proposal.py (62fe325)

git show --remerge-diff shows a single conflict hunk.

  • Main (flow: mine a dialog's empty-result guard; business-rule qualification cases #1617) changed the folded-step guard loop. A guard that only repeats
    an earlier step's exact text check goes into carried_guards; any other
    guard or wait rule is still refused at once. Main then confirms, after the
    prechecks are built, that each carried guard's text is among the precheck
    reads (:823-832).
  • Main's others block was context only in that change, not modified. This
    branch had already replaced it with _span_writes and writes= (reviewed
    earlier).
  • The resolution keeps main's carried-guard confirmation, then
    _span_writes (:834-838). Each returns a refusal on failure, and the spec
    is built only after both pass.
  • Order: the two checks read different inputs (guards and precheck text
    versus network lines), so order can't let a proposal past either one.
    Nothing from main's others and other_writes_seen was dropped that this
    branch hadn't already deliberately replaced.
  • Non-blocking note A: flow: mine a dialog's empty-result guard; business-rule qualification cases #1617 relies on the pre-check read running "right
    before the save". In this branch the prechecks run in prepare(), before
    the span's step writes go out in send() (web_form.py:851 vs :972).
    If a step write sits between the opener's check and the guarded step, the
    GUI path would check the guard after that write, but the API step checks
    it before. That only matters if the step write changes the duplicate-search
    result at run time when it didn't in the recording, and the scratch run
    checks the page's script writes. So the risk is low and not a blocker. A
    follow-up can refuse a carried guard when a role="step" write comes from
    a step between the precheck's from_step and the guarded step, or rerun
    the precheck right before the save.

3. The end-of-run rewrite

_halt_on_late_form_submit (replayer.py, about line 13300) replaces
last.error and sets failure_category = "safety_halt" whenever the check
is still armed and has a refusal.

  • When the span halted on the earlier save's refusal (the case the fix
    adds), the rewrite says the same thing in other words. It is cosmetic. The
    test accepts either wording.
  • It can hide a different halt reason. This needs a stray page submit
    during the span (after the read at the span's start), and the span halting
    or finishing for another reason. The rewrite then overwrites the span's own
    message, for example "the API step began sending … may have written;
    reconcile by reading the records back", a refuted record check, or a
    governed_refusal. It replaces them with "after an earlier save, …".
  • Fail-closed holds. The run is already unsuccessful (or becomes so),
    delivery_attempted and effect_results are kept, and the durable
    checkpoint was recorded before this pass with the original error. So no
    write is enabled and nothing is retried. The cost is that the report loses
    the reconcile guidance and the original category.
  • Non-blocking note B: append rather than replace. If last.error is set,
    use f"{last.error}; also after an earlier save, {late}", and keep an
    existing failure_category. This is a one-line follow-up.

Housekeeping

  • Logs: /private/tmp/claude-501/-Users-abrichr-oa-src-OpenAdapt--claude-worktrees-pr-backlog-9cdc67/ce7e8da1-8060-4a00-bb9e-a696c4171bf6/scratchpad/pr1540-prefix.log
    and /private/tmp/claude-501/-Users-abrichr-oa-src-OpenAdapt--claude-worktrees-pr-backlog-9cdc67/ce7e8da1-8060-4a00-bb9e-a696c4171bf6/scratchpad/pr1540-head.log.
    These are scratchpad files and are purged on reboot.
  • The pr1540 worktree is untouched and clean at 62fe325.

@abrichr
abrichr merged commit 197b3c9 into main Nov 3, 2026
45 checks passed
@abrichr
abrichr deleted the claude/api-step-page-script branch November 3, 2026 23:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant