Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 10 additions & 12 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,12 @@ backend owns a structured layer (a browser DOM, a native UIA/AX tree) the
resolution ladder's TOP rung re-finds the recorded target as an ELEMENT and
acts on it deterministically (`StructuralActionBackend`, see Resolution ladder).
Structure is preferred where present; the visual ladder is the fallback floor
for pixel-only substrates (RDP/Citrix/canvas). Backend evidence varies and is
stated honestly: the **shipped, end-to-end-exercised** backend is
Playwright-driven (headless-capable, CI-friendly, permission-free) and is the
only path proven against a real third-party app. Beyond it, a `WindowsBackend`
(UI Automation over the WindowsAgentArena server) is **proven structurally on a
local Windows-on-ARM VM** (record → compile → replay, DB-judged), and a FreeRDP
`RDPBackend` plus a Citrix/remote-display pixel-only backend exist as
**spikes, not validated integrations** — their live behavior is unmeasured to
the degree disclosed in `docs/backends/RDP.md` and `docs/desktop/CITRIX_PIXEL.md`.
They are adapters onto the one protocol, not rewrites.
for pixel-only substrates (RDP/Citrix/canvas). Playwright, Windows UIA, macOS
AX, Linux AT-SPI, RDP, and Citrix/pixel-window adapters implement the same
backend contract. Their evidence is bound to exact fixtures, counted tasks,
environments, or deployment qualifications in `docs/PRODUCT_STATUS.md` and
`docs/VERIFICATION.md`. A backend name alone does not show that an application
or environment is qualified.

## Core contracts (additive-only; do not change without updating this doc)

Expand Down Expand Up @@ -371,8 +367,10 @@ referral → New Encounter → click "Triage" → click Note field → type note
`Recorder` so frames/events are captured (before frame, act, wait settle,
after frame).

`PlaywrightBackend(page)` implements `Backend` (chromium, fixed viewport
1280x800, deviceScaleFactor=1). `Recorder(backend, out_dir)` wraps a backend
`PlaywrightBackend(page)` implements `Backend` for Chromium. The MockMed
fixture starts at 1280x800 with `deviceScaleFactor=1`. An attached recording can
start a new exact geometry epoch after a stable viewport or device-scale
change. `Recorder(backend, out_dir)` wraps a backend
with the same action methods plus `type_text(text, param=None)` and
`finish() -> recording dir`.

Expand Down
23 changes: 12 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@
[Discussions](https://github.com/OpenAdaptAI/openadapt-flow/discussions) ·
[Contributing](CONTRIBUTING.md)

**openadapt-flow is the OpenAdapt engine: a governed demonstration compiler.**
Record a task once, compile it to a deterministic program, and replay that
program deterministically with zero model calls on the healthy path. Instead of
silently doing the wrong thing when an interface drifts, it re-resolves from the
evidence the demonstration retained, or it **halts** for a human or an AI, gated
by an identity check and independent effect verification. It runs entirely on
your machine; nothing egresses unless you opt in.
**openadapt-flow is the demonstration compiler and governed runtime behind
OpenAdapt.** It compiles a demonstrated GUI workflow into a deterministic,
locally executable program. Healthy runs make no model calls. When an interface
drifts, Flow re-resolves from retained evidence. A person or configured model
can propose a repair. Identity, effect, and policy checks still apply, and Flow
halts when verification fails. It runs on your machine and doesn't send data
anywhere unless you opt in.

**Product state:** Flow enters Production only through an active signed,
expiring, and revocable release admission. If the admission is missing,
Expand Down Expand Up @@ -113,10 +113,11 @@ application itself never calls, so the screen cannot influence it. It ends
lies: the server rejects the write *after* the application has painted its
success banner, so every on-screen check passes while nothing lands. The
independent read of the system of record refutes the mined `record_written`
contract and the engine **HALTS** at the consequential step instead of
believing the screen. The caught fault's evidence is a clearly-labeled local
`run-broken/REPORT.md`; no shareable receipt is emitted for it, because only
`VERIFIED` runs may use the success rail.
contract. Because delivery reached the consequential step, the runtime returns
`RECONCILIATION_REQUIRED` and makes no blind retry or replay dispatch. The
caught fault's evidence is a clearly labeled local `run-broken/REPORT.md`. No
shareable receipt is emitted because only `VERIFIED` runs may use the success
rail.

For a live walkthrough, perform the demonstration yourself and then watch the
compiled replay at a visible pace:
Expand Down
18 changes: 10 additions & 8 deletions claims.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@ claims:
# ------------------------------------------------------------------ web
- id: web-supported
claim: >-
Web (browser) workflows are supported today: record a GUI workflow once,
then replay it deterministically and locally.
The Playwright browser path records a GUI workflow and replays it
deterministically and locally. Required CI exercises the recorder,
compiler, and replay contract.
surfaces: [README.md, website, docs]
tier: supported
evidence:
Expand Down Expand Up @@ -199,17 +200,18 @@ claims:
reflected evidence from Python at the settled boundary instead.
caveats:
- >-
"Supported" is scoped to the reference headless-browser backend in this
registry. Desktop and remote-display workflows use the separately scoped
acceptance and code-qualified claims below.
`supported` names the registry's required-CI evidence tier. It is not a
product state or a general application claim. Production requires active
release admissions for all seven product targets. A Production run also
needs an active workflow admission for the exact sealed bundle.
- >-
The full record->compile->replay browser suite runs in the required
e2e-browser PR gate and repeats in the weekly compatibility matrix.
- >-
Existing-session attachment is Chromium-only and loopback-only. It
requires a dedicated browser process started with remote debugging.
It does not claim support for the Capture Chrome extension prototype
or direct extension replay.
requires a dedicated browser process with remote debugging. The
Playwright-native browser path owns this claim. Capture extension code
is outside its evidence scope.

# -------------------------------------------------- deterministic $0 replay
- id: deterministic-zero-model-replay
Expand Down
6 changes: 3 additions & 3 deletions docs/LIMITS.md
Original file line number Diff line number Diff line change
Expand Up @@ -314,12 +314,12 @@ halt does not prove earlier actions were harmless.

## Interaction and environment limits

The recorded-surface evidence is strongest when the target remains inside one
captured browser surface and the demonstration exposes observable outcomes.
Qualification is strongest when the demonstration retains exact observations
and exposes independently verifiable outcomes.

| Condition | Current boundary |
| --- | --- |
| Zoom, DPI, font, layout, or viewport changes | Structural evidence can survive some reflow and visual rungs can survive some movement, but support is workflow-specific. Large rescale or reflow can halt. |
| Zoom, DPI, font, layout, viewport, or monitor changes | Browser attach recording accepts a stable viewport or device-scale change between actions, starts a new exact geometry epoch, and refuses an action that overlaps the transition. Native offline conversion can normalize a stable source-window move, resize, monitor change, or scale change into one fixed output viewport when every timeline row carries valid mapping metadata. It refuses a malformed timeline or a changed output viewport. RDP and Citrix live backends can rebaseline geometry between actions and refuse a change during the exact-frame lease. Other rescale or reflow behavior remains workflow-specific. |
| Native select menus, file choosers, permission prompts, and secure desktops | OS or browser chrome may not appear in the captured surface and may be unrecordable or undrivable. Prefer an application-level or keyboard/API path. |
| New tabs and windows | Opening a tab can be observed structurally where the backend supports it; interaction inside additional windows and multi-window coordination are not a general supported path. |
| Drag and drop or gesture-heavy controls | Not a general supported primitive. Use a structured/API alternative or validate a purpose-built workflow. |
Expand Down
10 changes: 5 additions & 5 deletions docs/PRODUCT_STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,16 +47,16 @@ product lifecycle state.
| AI-assisted repair | **Required CI contracts; deployment evidence required** | Local and remote VLM contracts, egress gates, refusal behavior, and retention boundaries are tested. | It is off by default. A model cannot authorize an action or prove identity or effect. The exact endpoint and task require workflow qualification. |
| Human teaching (`teach`) | **Required CI contracts; field evidence required** | Halt-to-correction-to-guarded-promotion and regression refusal run in default CI. | Evidence is controlled and synthetic. Broad authoring UX and field recovery time require deployment evidence. |
| Windows UIA replay | **Counted task acceptance** | Candidate `20260717-candidate-56759c8-v2` completed 3/3 exact WinForms trials with independently confirmed SQLite effects and 12 native UIA delivery receipts. Stale and ambiguous targets each refused 3/3; silent incorrect successes, over-halts, and model calls were zero. See [`benchmark/windows_uia/results.json`](../benchmark/windows_uia/results.json). | Acceptance covers the in-tree WinForms workflow and exact Windows VM. Each third-party application is qualified against its own controls, versions, identity rules, and effect oracle. |
| Desktop recording (`windows` / `macos` / `linux` / `rdp` / `citrix`) | **Required CI plus substrate acceptance** | `openadapt-capture` conversion, compile, and replay orchestration run in CI for every desktop selector, and the native substrate qualifications below prove the corresponding actuation paths. | Offline pixel capture cannot reconstruct structural accessibility evidence. Workflows that require UIA, AX, or AT-SPI identity use a live structural observer or are re-armed against the qualified application before release. Regulated profiles require declared secret handling and fail-closed privacy configuration. |
| Native macOS desktop actuation | **Scoped acceptance** | Candidate `b1b61a5` completed 3/3 TextEdit replace-and-save trials with exact file-byte effects and refused a two-window ambiguous selector without changing either file. See the [accepted evidence adjudication](../benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json). | Acceptance covers TextEdit on one macOS 15.7.3 Apple Silicon host and active user session. Customer applications require workflow-specific qualification. |
| Desktop recording (`windows` / `macos` / `linux` / `rdp` / `citrix`) | **Required CI plus substrate acceptance** | `openadapt-capture` is the canonical native screen, mouse, keyboard, timing, window-scope, and media-capture component. Capture conversion, compile, and replay orchestration run in CI for every desktop selector, and the native substrate qualifications below prove the corresponding actuation paths. | Offline pixel capture cannot reconstruct structural accessibility evidence. A workflow that requires UIA, AX, or AT-SPI identity must retain a live structural observation or receive that evidence during qualification. Regulated profiles require declared secret handling and fail-closed privacy configuration. |
| Native macOS desktop actuation | **Counted task acceptance** | Candidate `b1b61a5` completed 3/3 TextEdit replace-and-save trials with exact file-byte effects and refused a two-window ambiguous selector without changing either file. See the [accepted evidence adjudication](../benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json). | Acceptance covers TextEdit on one macOS 15.7.3 Apple Silicon host and active user session. Customer applications require workflow-specific qualification. |
| Native macOS AX structured identity | **Counted task acceptance plus required CI** | The macOS backend implements the same structured-layer contract as the browser DOM, Windows UIA, and Linux AT-SPI backends: it records a stable AX locator, re-finds the UNIQUE element at replay, refuses ambiguous / truncated / scope-escaping enumeration instead of guessing, and returns structured text under a point. Headless unit CI covers record/locate/refuse; a live-AX TextEdit run produced real evidence ([AX identity adjudication](../benchmark/macos_native/ax_identity_20260720.adjudication.json)); the record→compile→replay conformance test asserts zero model calls on healthy replay. See [`tests/test_macos_structural.py`](../tests/test_macos_structural.py) and the [capability matrix](../tests/test_backend_capability_matrix.py). | The backend uses gated point-bound physical click after structural resolution rather than claiming AXPress everywhere. AX exposure varies by application; controls without durable AX identity use the visual ladder. |
| Native Linux desktop actuation | **Counted task acceptance plus required CI** | The required `linux-atspi-x11` job runs a real GTK3 application against AT-SPI inside an isolated Xvfb/session-D-Bus environment: 3 clean exact-file-effect trials, 3 ambiguous-target refusals, and 3 stale-target refusals. Unit CI covers the remaining window, traversal, capture, physical-input, and portal boundaries. | Acceptance is bounded to the in-tree GTK3 workflow and CI image. Each application and environment retains its own qualification. The built-in driver uses X11; Wayland requires a live operator-approved XDG portal session and refuses without one. |
| RDP | **Counted task acceptance plus required CI** | Candidate `82a658a` completed 3/3 real-network Aardwolf RDP trials into Windows 11, with a guest-tools file oracle, zero failures, zero silent incorrect successes, zero over-halts, and zero model calls. The public multi-window FreeRDP campaign adds a bounded 27-trial contract with independent SQLite, CSV, and Maildir oracles. The backend also rebaselines a changed framebuffer between actions, refuses a change during the exact-frame lease, refuses unsupported horizontal scroll before delivery, and classifies transport failures as uncertain delivery. See the [accepted batch](../benchmark/rdp/ACCEPTED_BATCH_82A658A.md) and [campaign contract](../benchmark/rdp_multiapp/README.md). | The accepted batch covers the tested 1280×800 transport/input task. The multi-window fixture uses synthetic applications. Target applications, identity/effect rules, session policies, and display conditions are qualified per deployment. A composite multi-monitor session remains deployment-qualified evidence, not part of the accepted 1280×800 batch. |
| Citrix / pixel-only remote display | **Required CI plus counted no-DOM stand-in** | `--backend citrix` binds an exact Citrix Workspace window, readiness marker, pixel-only ladder, governed run, durable resume, and report; required CI covers those orchestration and refusal contracts. The window driver recalculates capture scale after a resize or cross-monitor move and refuses DPI or geometry drift during input. The public real-ICA preflight adds distinct authority keys, executable and oracle attestations, a signed display and monitor-topology observation, explicit reliability metrics, one-use campaign state, crash recovery, and uncertain-dispatch handling. Separately, the retained no-DOM driver qualification passed 3 healthy effect-confirmed trials and 3 drift safe-halts with zero model calls, silent incorrect successes, or false completion, and records `code_readiness_accepted=true`. | The counted stand-in and preflight do not claim live ICA/HDX acceptance. A live result remains bound to the exact Workspace/server/application/display matrix, customer-approved executable, and independent effect oracle. Deployment-specific recipes, data, and thresholds stay outside the public repository. |
| Identity verification | **Required CI; exact workflow binding required** | Wrong-entity refusal and adversarial corpora run in CI. | Unarmed clicks have no identity check. Real compiled bundles currently arm only a subset of clicks. |
| Identity verification | **Required CI; exact workflow binding required** | Wrong-entity refusal and adversarial corpora run in CI. | An action without an identity contract has no entity check. Workflow admission must bind the exact armed actions and identity authority. |
| System-of-record effect verification | **Required CI; exact verifier binding required** | REST, FHIR, SQL, file, and document verifier contracts catch fault classes that screen-only verification misses. A deployment with multiple reviewed read boundaries selects and preflights the strongest evidence tier before input, retains that binding through durable resume, and never downgrades after an action. | Effects are not generally inferred; both authored effects and a configured verifier are required. A selected verifier that becomes unavailable halts or enters reconciliation. |
| Lint and certification policies | **Required CI** | The CLI reports coverage gaps and refuses bundles that violate a selected policy. | Certification is opt-in; `replay` remains the permissive tutorial path. Use fail-closed `run` for a deployment. |
| Durable pause, approval, and resume | **Required CI; authenticated operator route required** | Checkpoint, bundle-version binding, approval, stale-pause, and resume semantics are tested. | Operator identity is recorded, not integrated with an enterprise IdP; field operation is unmeasured. |
| Lint and certification policies | **Required CI** | The CLI reports coverage gaps and refuses bundles that violate a selected policy. | `replay` remains the permissive tutorial path. Governed deployment uses fail-closed `run` plus the required release and workflow admissions. |
| Durable pause, approval, and resume | **Required CI; authenticated operator route required** | Checkpoint, bundle-version binding, approval, stale-pause, and resume semantics are tested. | The engine records an asserted operator identity. Desktop, Cloud, or a customer-local identity route must authenticate that principal. |
| Typed business decisions | **Required CI; authenticated operator route required** | A typed qualification API adds or updates a finite decision node without manual manifest edits and invalidates stale certification. The graph runtime pauses at the certified choice, validates a supplied principal and role, retains a signed durable receipt, restores it after a crash, revalidates the live application, and permits only the certified successor branch. | The engine does not authenticate a user. Desktop, Cloud, or a customer-local identity route must supply an authenticated principal. A decision never replaces entity identity or effect verification. |
| Reviewed judgment cases | **Required CI contracts; reviewed workflow evidence required** | Qualification binds typed facts, local evidence hashes, reviewer provenance, and the exact decision contract to reviewed examples or counterfactuals. It preserves permanent human authority, requires reciprocal contrasts for an automatic-rule candidate, and refuses certification when a case still needs evidence. | The case layer does not synthesize executable policy from one or more examples. A reviewed automatic rule must be authored and qualified through the normal program path. |
| Qualified remote decision tasks | **Required CI contracts; negotiated peer schema required** | An explicitly negotiated V2 task binds optional reviewed entity wording to the exact qualification, bundle, step, policy, and pause. V1 stays byte-compatible, and an unavailable or unrecognized class renders as the signed neutral `record` or `item` fallback. | V2 requires `openadapt-types` 0.10.x and a consumer that negotiates the schema. Actual entity identifiers and live revalidation stay inside the customer-controlled runner. |
Expand Down
Loading