Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
732fdcb
feat(flow): interruption classes cleared only when provably read-only
abrichr Nov 4, 2026
4e99647
feat(flow): classify the whole interruption and check where a click l…
abrichr Nov 4, 2026
a7672c3
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 8, 2026
01bc517
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
389f10e
fix(flow): check the clicked control is the declared dismiss control
abrichr Nov 9, 2026
776afac
docs: refresh llms-full.txt
abrichr Nov 9, 2026
d26d722
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
da48f87
fix(flow): keep the dismiss control's label for the whole click
abrichr Nov 9, 2026
9c0e96d
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
edb08b7
fix(flow): count a CSSOM relabel of the dismiss control as a change
abrichr Nov 9, 2026
7acb41c
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
223711d
fix(flow): refuse a click guard that can't observe the whole label
abrichr Nov 9, 2026
71b34b2
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
1cfff12
fix(flow): block the input when the click guard's check can't finish
abrichr Nov 9, 2026
a5313ce
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
c1508e0
fix(flow): keep the click guard from reading any inherited property
abrichr Nov 9, 2026
9aa61a0
Merge origin/main into claude/interruption-recovery
abrichr Nov 9, 2026
d2f199f
Merge remote-tracking branch 'origin/main' into HEAD
abrichr Nov 9, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ monorepo and installable from source.
| Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) |
| Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](packages/flow/benchmark/pixel_scale/README.md) |
| Lists every change made before a stop and rolls it back with compensations a person approved once, then proves it by reading the whole database: equal to the read from before the run is `ROLLED_BACK`, anything else names the exact rows. | `undo`, `rollback`, [ROLLBACK.md](packages/flow/docs/ROLLBACK.md), [AFTER_A_STOP.md](packages/flow/docs/AFTER_A_STOP.md) |
| Clears a known interruption (a session warning, a banner, a tour) by itself, only when the dismissal is provably read-only: matched by its text and control, a closed set of non-committing labels, no write seen by the browser write fence or the system-of-record read-back, and the step's normal identity check afterwards. Anything else stops the run. Classes are per-app data. | [INTERRUPTION_RECOVERY.md](packages/flow/docs/INTERRUPTION_RECOVERY.md) |
| Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](packages/flow/docs/REPAIR_LIFECYCLE.md) |
| Repairs a change that needs new steps, such as a new required dialog or a control moved into a menu: up to three new steps, each with its own record check, probed up to the save before one person approves. | [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md), [#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325) |
| Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](packages/flow/docs/JIT.md) |
Expand Down
1 change: 1 addition & 0 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -539,6 +539,7 @@ monorepo and installable from source.
| Checks a save on desktop, RDP and Citrix apps with no API at all, by looking the record up in the app's own screens in a separate session. | [EFFECT_KIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/EFFECT_KIT.md#separate-session-desktop-read-back-desktop_readback) |
| Keeps pixel-path runs working across display scaling from 100% to 175% and font smoothing on or off, and stops rather than saves when the record changed. | [benchmark/pixel_scale](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/pixel_scale/README.md) |
| Lists every change made before a stop and rolls it back with compensations a person approved once, then proves it by reading the whole database: equal to the read from before the run is `ROLLED_BACK`, anything else names the exact rows. | `undo`, `rollback`, [ROLLBACK.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/ROLLBACK.md), [AFTER_A_STOP.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/AFTER_A_STOP.md) |
| Clears a known interruption (a session warning, a banner, a tour) by itself, only when the dismissal is provably read-only: matched by its text and control, a closed set of non-committing labels, no write seen by the browser write fence or the system-of-record read-back, and the step's normal identity check afterwards. Anything else stops the run. Classes are per-app data. | [INTERRUPTION_RECOVERY.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTERRUPTION_RECOVERY.md) |
| Repairs itself when the app changes: governed repair, agent-demonstrated fixes, fleet patches, and self-repair probed up to the save. | [REPAIR_LIFECYCLE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/REPAIR_LIFECYCLE.md) |
| Repairs a change that needs new steps, such as a new required dialog or a control moved into a menu: up to three new steps, each with its own record check, probed up to the save before one person approves. | [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md), [#1325](https://github.com/OpenAdaptAI/OpenAdapt/pull/1325) |
| Compiles a computer-use agent's verified run, with no human demonstration needed. | [JIT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/JIT.md) |
Expand Down
4 changes: 4 additions & 0 deletions packages/flow/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

## Unreleased

### Features

- Interruption classes: a run clears a known interruption, such as a session warning, a banner or a tour, by itself, but only when the dismissal is provably read-only. The class must match by its text and control, the control's label must be in a closed non-committing set and be the live label of the control that is clicked, held for the whole click by a page-side click guard, and no write watcher may see a write. On the web the watcher is the browser write fence; on any backend, including desktop and RDP, it is the system-of-record read-back. Afterwards the step's normal identity check must pass. Classes are per-app data (`runtime.interruptions_path`) or learned from a halt (`learn_interruption`). Each dismissal is in the run report. See `docs/INTERRUPTION_RECOVERY.md`.

### Changed

- `scaffold-verifier` closes three gaps the zero-config benchmark found. A generated REST read now carries `records_complete: true` when `--complete-read` declares it or one unpaged response was seen to hold the whole collection; otherwise it stays incomplete and `verify-run` returns INDETERMINATE. A step counts as a write when the capture shows the page sending a POST, PUT, PATCH or DELETE to its own origin, or its store snapshots differ, not only when its label looks like Save. `--probe-reads` (with optional `--probe-base`) finds a read for an app that never reads the saved record back: it probes the collection and item reads the write implies with GET on the local machine, uses one only when it returns the written values and nothing outside it could change unseen (otherwise it refuses and writes the TODO draft), and records every probe (`read_discovery` in the proof, or `read_discovery.json` when none confirms).
Expand Down
212 changes: 212 additions & 0 deletions packages/flow/docs/INTERRUPTION_RECOVERY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
# Interruption recovery

Production apps interrupt a run: a session warning, an "Update available"
banner, a feature tour, a notification, a modal from background polling.
Without help, each one stops a compiled run for a person. An *interruption
class* lets the run clear a known interruption itself, but only when the
dismissal is provably read-only. Anything else stops the run as before.

## The policy

A run clears an interruption only when all four of these hold:

1. **It matches a class by identity, on the whole dialog.** Every text the
class lists is on screen, and the dismiss control is found by its role,
accessible name, selector, automation ID or window. Position is never
used. The button word never decides: the backend reads the whole
interruption (its full text, accessible title and owning window). The
run halts unless all of these hold:
- the dialog shows every class text;
- it carries the class's title and window, if the class declares them;
- it holds no business word (`BUSINESS_WORDS`: a patient, a record, a
chart, an order, or saving, applying, changing, discarding, deleting
or confirming anything);
- it says at most 240 characters beyond the class texts and the
control's label.

"Keep my changes? [Yes]" and "Apply template to patient? [OK]" can look
like "Stay logged in? [OK]", but they never match a class. A backend
that can't read the whole dialog can't prove the class, so the run
halts. This applies to pixel surfaces, and to native desktop backends
until they implement `interruption_text`.
2. **The dismissal does not commit anything.** The dismissal is Escape, or a
control whose label is in a closed set: close, dismiss, not now, maybe
later, skip, got it, continue session, stay signed in, and similar
(`NON_COMMITTING_LABEL` in `openadapt_flow/interruptions.py`). "OK",
"Continue", "Yes", "Accept" and any write word are outside the set, so a
class that uses one is refused when it loads.

The label checked is the label of the control that gets clicked. A
control class declares its role and accessible name, and a class without
both is refused when it loads. A selector or automation id may resolve to
any control, so before the click, and again at the last moment before
input, the backend must show that the resolved element is the one
control with the declared role and exact accessible name. If the class
says "Close" and its selector finds "Sign out", the run halts with no
click. So does a backend that can't read the live control
(`structural_control_is`), which today means every backend except the
browser.

The label must also hold for the whole click. In the browser, a click
guard is installed at the start of every document, ahead of any page
script, so its capture-phase listeners on `window` run before any page
listener. It is dormant until a dismissal arms it on the bound control.
While armed, it watches the control, the nodes its `aria-labelledby` and
`aria-describedby` name, its labels, and the page's stylesheets. At
every press and click it also re-reads how each of them renders: the
computed `::before` and `::after` content, their own computed content,
and `display` and `visibility`, so a stylesheet rule added through the
CSSOM (`insertRule`, `adoptedStyleSheets`) that relabels or hides the
text counts as a change. This evidence must be whole: if the control or
any node that labels it has more than 64 nodes in its subtree (itself
included, `LABEL_GUARD_MAX_NODES`), or the walk can't finish, the guard
refuses to arm and nothing is clicked. A label that grows past that
limit during the click is a change. The guard reads all of this only
through the browser's own getters and methods, captured before any page
script runs, so a getter or method the page defines later never runs.
Every dictionary it hands to a built-in (a property descriptor, observer
options) has no prototype, and every list a built-in iterates brings its
own iterator, so no inherited property is ever read. A test traps
`Object.prototype` keys and the built-ins the guard could reach, and
checks that no call from the guard's script reaches them.
An input is let through only when the whole check finishes and passes:
an error anywhere in it blocks the input and counts as a change. If any
of them changed before a press or click arrives, or the input is not on
the control, the guard cancels that input before the page sees it. The
run then halts with the dismissal reported as delivered and its outcome
uncertain (`LabelChangedDuringClick`), so it is reconciled, never
retried. A label that turns "Close" into "Sign out" as the pointer
arrives is caught this way. A document that loaded before the guard was
installed, a guard that can't arm, or one whose listeners are gone after
the click (for example, erased by `document.open`) halts too. Learning a
class from a halt uses the same guard.

The report agrees with the fence. Every request made while the fence is
up is listed in the dismissal's `requests_seen`, reads included, and a
write in that log that no write attempt names is added as one. So a
report never says "no write attempts" while the fence saw a write.
3. **No watcher sees a write.** The dismissal runs under every write watcher
the run has (`openadapt_flow/runtime/effect_watch.py`):
- in a browser, the write fence. It is the same hold that guards a
checked save, with nothing allowed through. Every write request, new
WebSocket, worker and socket message is stopped before it leaves the
page, and each is reported.
- on any backend, the system-of-record read-back that the run's writes
are checked with. It is read before and after the dismissal; any
difference is a write. This is the watcher for desktop and RDP or
Citrix runs, which have no network to fence.

A run with no watcher, a fence that can't go up, or a read-back that
can't be read does not dismiss. A dismissal that a watcher saw write
stops the run as a safety halt.
4. **The expected screen comes back.** After one dismissal the screen must
settle, the class's text must be gone, and the step then resolves its
target through its normal identity check. If the screen isn't the one
the step expects, the run stops there.

A run makes at most one dismissal per class per step, and never retries one.

## A popup that steals the click

A toast or modal can appear after the last check and before the click
lands, so the input reaches the popup instead of the target. On the web,
every structural click is checked twice:

- **Before the input.** Playwright's actionability trial and the
existing target and context checks refuse a covered target before any
input is sent.
- **After the input.** A witness armed just before the click records,
from the first moment the page sees each trusted pointer event,
whether it reached the verified element. If any part of the click
(press, release or click) reached something else, the outcome is
reported as delivery uncertainty (`InputLandedElsewhere`). The run then
stops and reconciles; it never retries. If the click navigated away,
the witness is gone with the old document, and the checks before the
input stand alone.

The same check after dispatch on native desktop and RDP/Citrix backends
(the foreground window, the focused element, and a frame diff of the
target region) is not built yet.

## The record

Each dismissal is recorded in the step's `interstitial_actions` in the run
report (`report.json`, which the run seal signs). The record includes the
class, the control's label, the frames before and after, the watchers that
were up (`effect_watchers`), and any write they saw (`write_attempts`).

## Declaring classes

Classes are data. The engine has no rule for any particular app. Put a
per-app file next to the deployment and point `runtime.interruptions_path`
at it:

```yaml
interruptions:
- name: session expiry warning
texts: ["Your session will expire"]
dismiss: {control: {role: button, name: Stay signed in}}
- name: update banner
texts: ["Update available"]
dismiss: {control: {role: button, name: Close}}
- name: product tour
texts: ["Take a quick tour"]
dismiss: {key: Escape}
```

`texts` must all be present. `dismiss.control` takes `role`, `name`,
`selector`, `automation_id` and `window`. `gone` names the text whose
absence clears the interruption; it defaults to the first text. If any
entry fails the policy, the run refuses to start.

## Learning a class from a halt

When a person clears a new interruption during a halt,
`openadapt_flow.interruptions.learn_interruption` can learn it as a class.
It builds the class from the interruption's texts and the control the
person chose, and checks that class against rules 1 and 2. It then clicks
the control on the live screen under the run's watchers. It keeps the class
only when no watcher saw a write and the text is gone. If the dismissal
writes, the class is not learned. `save_interruption_class` adds the class
to the app's file.

## Cases

These cases were registered before the runs. The real-browser cases are in
`tests/test_interruptions_browser.py`, which uses headless Chromium, the
real replayer and OCR, and a local synthetic app that saves with
`POST /save`. The fake-backend cases are in `tests/test_interruptions.py`.

| Case | Expected | Result |
|---|---|---|
| Session warning, "Stay signed in" (sends only a GET) | cleared, save lands | passes |
| "Update available" banner, "Close" | cleared, save lands | passes |
| Session warning whose "Stay signed in" POSTs /logout | safety halt; the logout is stopped in the browser, nothing saved | passes |
| Unknown dialog | halt, no click | passes |
| Session text plus "Apply template to patient?" behind "Stay signed in" | halt before any input | passes |
| A popup covers Save as the pointer arrives | refused before any input (pre-dispatch trial) | passes |
| A popup covers Save between press and release | delivery uncertainty `InputLandedElsewhere`, no retry, nothing written | passes |
| Dialog mentions "Keep my changes?" or "Unsaved record" | halt before any input | passes |
| Backend can't read the whole dialog | halt before any input | passes |
| Dialog says much more than the class texts; title mismatch | halt | passes |
| Class whose control reads "OK", "Continue", "Accept", "Confirm" | refused at load | passes |
| Class found only by position | refused at load | passes |
| Desktop run: Escape, read-back unchanged | cleared (`record` watcher) | passes |
| Read-back changes during the dismissal | safety halt | passes |
| Run with no watcher; fence refused | no dismissal, halt | passes |
| Dismissal leaves a different screen | halt at the step's identity check | passes |

## Limits

- A `GET` request with a side effect gets through the browser fence. On
desktop backends, the read-back sees only the records that the run's
verifier reads. Configure a read-back that covers the records the run
touches.
- A session keep-alive that POSTs is a write, so the run halts on it. Use
a session warning whose control only closes it, or re-authenticate.
- Consent prompts (cookie "Accept" or "Reject") are decisions, not
dismissals, so they stay outside the closed set.
- Session re-authentication, a bounded re-resolve after late page loads,
the native continuity check after dispatch, and real-app runs on
OpenEMR and Dolibarr (including behind a chaos proxy) are not covered
here.
Loading
Loading