From 558f5c8e71dca40e36a071f7a6321956a1c9643d Mon Sep 17 00:00:00 2001 From: abrichr Date: Tue, 3 Nov 2026 13:02:32 -0500 Subject: [PATCH 1/2] Docs: scope claims that drifted from the code - Pipelined read-back: durable runs can defer their last step's check. - Production-lifecycle 3.17x: a cost-model ratio with an assumed 3-minute hand entry and the draft #1516, not an observed saving. - Define "no model calls": local OCR and the optional local grounding model still run on the machine. - Fleet patches move as files or through a local hub; CloudPatchFeed is a stub. - induce --held-out and the learn loop's held-out check are structural coverage, not behavioural correctness. --- CAPABILITIES.md | 12 ++++--- README.md | 2 +- llms-full.txt | 31 ++++++++++++++++--- llms.txt | 8 +++-- packages/agent/README.md | 2 +- packages/flow/CHANGELOG.md | 2 +- packages/flow/README.md | 8 +++-- .../production-2026-10-25-final/README.md | 16 ++++++++-- .../production-2026-10-25-final/REPORT.md | 2 ++ packages/flow/docs/LEARN_FROM_MANY_RUNS.md | 6 ++-- packages/flow/docs/LIMITS.md | 17 ++++++++++ packages/flow/docs/REPAIR_LIFECYCLE.md | 4 ++- .../flow/openadapt_flow/learning/__init__.py | 5 ++- .../runtime/readback_pipeline.py | 10 ++++-- 14 files changed, 98 insertions(+), 27 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 1f9d0d7c2d..dd646607da 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -69,7 +69,9 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released. + run, and fleet drift patches shared with no patient data are released + (moved as files or through a local reference hub; there is no hosted + patch feed, and `CloudPatchFeed` is a stub that refuses). Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main, and so are fixes of up to three new steps. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task @@ -81,7 +83,9 @@ monorepo and installable from source. to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. 10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Personal and health data are redacted from text, + and send nothing out. Local OCR, and the local grounding model when it is + turned on, still run on the machine + ([what "no model calls" means](packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. @@ -106,7 +110,7 @@ monorepo and installable from source. | Has a written threat model for prompt injection against the agent that teaches or repairs a program, with tests. | [THREAT_MODEL.md](packages/flow/docs/THREAT_MODEL.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy, desktop and tray), each released from the monorepo. | [README](README.md) | -| Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](README.md) | +| Makes no model calls on healthy runs; local OCR can still read the screen on the machine. Model training (openadapt-ml) is separate research. | [LIMITS.md](packages/flow/docs/LIMITS.md#what-no-model-calls-means) | ## The full map @@ -263,7 +267,7 @@ improvising or retrying when something is off. | Repair a relabeled save button: prove from the recorded identity hash that only the label changed (same control id, kind and form), re-qualify the fix on the new release, and have a person admit the certified copy afresh, since a save button's identity is part of the record check | On main | Real app (local OpenEMR, synthetic data): after the save button was relabeled, the stopped item re-ran and saved once after 8.3 operator minutes and 763 s of re-qualification; 896 s from stop to done ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | `openadapt_flow/repair/relabel.py`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md#a-relabeled-control-proof-that-only-the-label-changed) | | A duplicate check that reads the app's match list (`no_duplicate_candidate`): stop only on a listed record with the item's last name or date of birth, fail closed on an unreadable or unloaded list | On main | Real app (local OpenEMR, synthetic data): false stops per 100 items 70.4 before, 16.4 after; 0 silent wrongs in 30 saves ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | [DUPLICATE_CHECK.md](packages/flow/docs/DUPLICATE_CHECK.md) | | A switched-off `problems_only` gate asks a person before the save instead of stopping the item | On main | Synthetic; real app in [#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513) | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#when-the-mode-turns-itself-off) | -| Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | +| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub. There is no hosted feed: `CloudPatchFeed` is a stub that refuses | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents diff --git a/README.md b/README.md index ac0d3e9daa..9866ab00f2 100644 --- a/README.md +++ b/README.md @@ -280,7 +280,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Binds each approval to exactly what will be saved**, with one-tap answers on a paired phone. - **Repairs itself when the app changes**, probing a fix up to the save and asking a person to approve it. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. -- **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. +- **Runs locally**: no model calls on healthy runs (local OCR can still read the screen; [what that means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)), patient data redacted, signed receipts. [CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. diff --git a/llms-full.txt b/llms-full.txt index 46b17ec00a..d9e671df5c 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -290,7 +290,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Binds each approval to exactly what will be saved**, with one-tap answers on a paired phone. - **Repairs itself when the app changes**, probing a fix up to the save and asking a person to approve it. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. -- **Runs locally**: no model calls on healthy runs, patient data redacted, signed receipts. +- **Runs locally**: no model calls on healthy runs (local OCR can still read the screen; [what that means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)), patient data redacted, signed receipts. [CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. @@ -466,7 +466,9 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released. + run, and fleet drift patches shared with no patient data are released + (moved as files or through a local reference hub; there is no hosted + patch feed, and `CloudPatchFeed` is a stub that refuses). Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main, and so are fixes of up to three new steps. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task @@ -478,7 +480,9 @@ monorepo and installable from source. to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. 10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Personal and health data are redacted from text, + and send nothing out. Local OCR, and the local grounding model when it is + turned on, still run on the machine + ([what "no model calls" means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. @@ -503,7 +507,7 @@ monorepo and installable from source. | Has a written threat model for prompt injection against the agent that teaches or repairs a program, with tests. | [THREAT_MODEL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THREAT_MODEL.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy, desktop and tray), each released from the monorepo. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | -| Makes no model calls on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | +| Makes no model calls on healthy runs; local OCR can still read the screen on the machine. Model training (openadapt-ml) is separate research. | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means) | ## The full map @@ -660,7 +664,7 @@ improvising or retrying when something is off. | Repair a relabeled save button: prove from the recorded identity hash that only the label changed (same control id, kind and form), re-qualify the fix on the new release, and have a person admit the certified copy afresh, since a save button's identity is part of the record check | On main | Real app (local OpenEMR, synthetic data): after the save button was relabeled, the stopped item re-ran and saved once after 8.3 operator minutes and 763 s of re-qualification; 896 s from stop to done ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | `openadapt_flow/repair/relabel.py`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md#a-relabeled-control-proof-that-only-the-label-changed) | | A duplicate check that reads the app's match list (`no_duplicate_candidate`): stop only on a listed record with the item's last name or date of birth, fail closed on an unreadable or unloaded list | On main | Real app (local OpenEMR, synthetic data): false stops per 100 items 70.4 before, 16.4 after; 0 silent wrongs in 30 saves ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | [DUPLICATE_CHECK.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/DUPLICATE_CHECK.md) | | A switched-off `problems_only` gate asks a person before the save instead of stopping the item | On main | Synthetic; real app in [#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513) | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#when-the-mode-turns-itself-off) | -| Fleet drift patches: a verified fix with no patient data, shared across sites under policy | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | +| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub. There is no hosted feed: `CloudPatchFeed` is a stub that refuses | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents @@ -1291,6 +1295,23 @@ saves resemble the labelled ones; it doesn't survive an application, clinic or form change. The only evaluation is on synthetic data. See [`DECISION_TRIAGE.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/DECISION_TRIAGE.md). +## What "no model calls" means + +"No model calls" and the `model_calls` count in a run report mean no request +to a language or vision model: no hosted model API and no model endpoint you +configured. It doesn't mean no machine-learned component runs: + +- Local text recognition (RapidOCR on ONNX Runtime by default, or the + operating system's own OCR with `OPENADAPT_FLOW_OCR=native`) reads the + screen on the machine wherever a healthy run needs screen text: identity + checks on pixels, landmarks and on-screen read-back. It is a neural network, but it runs in the process + and sends nothing. +- The optional local grounding model (`--local-grounding`, off by default) + runs on the machine when you turn it on. The run report counts it apart, + as `local_model_calls`. +- Model-assisted target finding and repair call a model only when you + configure one, and those calls are counted in `model_calls`. + ## What "deterministic" means The compiled program fixes the action sequence, parameters, target evidence, diff --git a/llms.txt b/llms.txt index ab23ba45fe..c82816d438 100644 --- a/llms.txt +++ b/llms.txt @@ -51,7 +51,9 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released. + run, and fleet drift patches shared with no patient data are released + (moved as files or through a local reference hub; there is no hosted + patch feed, and `CloudPatchFeed` is a stub that refuses). Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded @@ -60,7 +62,9 @@ monorepo and installable from source. to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. 10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Personal and health data are redacted from text, + and send nothing out. Local OCR, and the local grounding model when it is + turned on, still run on the machine + ([what "no model calls" means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. diff --git a/packages/agent/README.md b/packages/agent/README.md index 3b65776b02..7e9c674354 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -9,7 +9,7 @@ Your AI agent decides what to enter. OpenAdapt enters it in the app and checks t Use it if your agent reads referrals, intake forms, or authorizations and works out what belongs in the record, but a person still types the result into an app with no usable API. Examples are an electronic medical record (EMR) and a payer portal. This package is the tool your agent calls to do that typing. -- A person or your agent shows OpenAdapt the task in the real app. OpenAdapt replays those steps, and by default a run makes no AI model calls. +- A person or your agent shows OpenAdapt the task in the real app. OpenAdapt replays those steps, and by default a run makes no AI model calls. Local text recognition (OCR) can still read the screen, on your machine. - Before it types, OpenAdapt checks that the right record is open. After it saves, it reads the record back. - Your agent gets one of four outcomes: done and checked, stopped before saving, check the record, or didn't run. When OpenAdapt can't confirm a save, it says so, and a person checks the record. - It runs on your customer's computer, one that can already open the app. Your agent, or a small program acting for it, has to run on that computer too, since it starts OpenAdapt and talks to it directly. If your agent runs in your own cloud, ask us about a hosted route. diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index 33467f5c83..5d49a3b3c2 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -115,7 +115,7 @@ A person's approval counts only when a registered approver key signed it, and th - Under `problems_only`, problems with an item's own input leave the switch on and send that item to a person; program and environment problems turn it off, and a person re-arms it with one signed tap ([#1517](https://github.com/OpenAdaptAI/OpenAdapt/pull/1517)). - Look-alike rows are counted on Windows, macOS, Linux and pixel-only surfaces, so a step that picks a row halts when it cannot tell the target from its twin ([#1504](https://github.com/OpenAdaptAI/OpenAdapt/pull/1504), [#1518](https://github.com/OpenAdaptAI/OpenAdapt/pull/1518), [#1557](https://github.com/OpenAdaptAI/OpenAdapt/pull/1557)). - `openadapt-agent serve --teach-browser` lets a coding agent turn a read-only browser task into a named tool it calls again with new values; `openadapt setup` registers it with Claude Code, Cursor, Codex and Gemini CLI ([#1502](https://github.com/OpenAdaptAI/OpenAdapt/pull/1502)). -- `induce --held-out` scores each held-out item by running it and checking the record independently ([#1527](https://github.com/OpenAdaptAI/OpenAdapt/pull/1527)). +- `openadapt_flow.validation.behavioral_generalization` scores each held-out item by running it and checking the record independently. `induce --held-out` is unchanged: it prints structural trace coverage and runs nothing ([#1527](https://github.com/OpenAdaptAI/OpenAdapt/pull/1527)). ### Fixes diff --git a/packages/flow/README.md b/packages/flow/README.md index 5c93a5a09d..e8025dfeef 100644 --- a/packages/flow/README.md +++ b/packages/flow/README.md @@ -376,7 +376,10 @@ claude --mcp-config mcp.json # then ask: Add an 18.75 expense for parking Recording, compiling, replaying, and production runs all happen on your machine. A healthy run makes no model calls, and model-assisted target finding -is off unless you turn it on in the CLI or deployment config. That isn't the +is off unless you turn it on in the CLI or deployment config. Local OCR can +still read the screen on your machine, and so does the local grounding model when +you turn it on; see +[what "no model calls" means](docs/LIMITS.md#what-no-model-calls-means). That isn't the same as no network: the app you drive and the record check still talk to their own services. @@ -449,7 +452,8 @@ patients. The systems each result ran on, and the boundary of every claim: - [docs/BENCHMARKS.md](docs/BENCHMARKS.md): every benchmark, with method and caveats. - [Fleet drift patches](docs/FLEET_DRIFT_PATCHES.md): opted-in installs that - automate the same application share signed repairs for the same UI change. + automate the same application share signed repairs for the same UI change, + moved as files or through a local reference hub (there is no hosted feed). Each tenant tests a patch on its own cases before it is used. - [docs/REWARD_WORKER.md](docs/REWARD_WORKER.md): `serve-reward` scores a model's training episode by reading the record after it ends. diff --git a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md index b088e817a9..6e33c93014 100644 --- a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md +++ b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md @@ -7,6 +7,14 @@ seed 7, 10 simulated days, workload `lifecycle-workload/2026-10-24.1`, and the save button relabeled before item 26. This time every fix is on main except the record-channel duplicate check (#1516, still a draft). +> **Scope.** The 3.17x below is a modelled ratio, not an observed saving. +> Operator minutes come from a cost model (approval 20 s, stop 90 s, +> reconcile 120 s, repair review 120 s), and hand entry is assumed at 3 +> minutes per item; no person was timed. The build includes the draft +> record-channel duplicate check (#1516), which is not on main. It ran on a +> local OpenEMR with synthetic patients and a scripted operator. Don't quote +> it as a production throughput gain. + ## What was measured Branch `measure/production-lifecycle-final` at **8bfd7f6af** (`run.log` @@ -41,8 +49,8 @@ At x1 of the cost model (approval 20 s, stop 90 s, reconcile 120 s, repair review 120 s, plus a 30 s interrupt on every request). These are run-only operator minutes. -**Headline: 1.055 verified items per operator-minute, 3.17 times the by-hand -rate at 3 minutes per item.** That is 51 verified items over 48.33 operator +**Headline (cost model): 1.055 verified items per operator-minute, 3.17 times +the by-hand rate if hand entry takes 3 minutes per item.** That is 51 verified items over 48.33 operator minutes, or 0.956 when the 5.0 setup minutes are included. | By hand, minutes per item | Verified items per operator-minute | By hand per operator-minute | Ratio | @@ -93,7 +101,9 @@ Phase 0 has three criteria: at least 2 times hand entry at 3 minutes per item, 0 silent wrongs, and at least 15 of 17 independent faults caught. This run measures the first two: -- **Throughput: met.** The run reached 3.17 times hand entry. +- **Throughput: met under the cost model.** The run reached a modelled 3.17 + times hand entry at an assumed 3 minutes per item, with the draft #1516 + included. No by-hand time was measured. - **Silent wrongs: met.** There were 0 in the 41 items claimed done, judged by direct SQL and never by the product's verdict. - **Fault coverage: measured separately** (#1539, the independent fault diff --git a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md index f341dc82d8..49baeb3df9 100644 --- a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md +++ b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md @@ -1,5 +1,7 @@ # Lifecycle soak report +> Scope: operator minutes come from a cost model, hand entry is assumed at 3 minutes per item (no person was timed), and the build includes the draft duplicate check #1516. The ratio is modelled, not an observed production saving. + Protocol `lifecycle/2026-10-20.1`, workload `lifecycle-workload/2026-10-24.1`, seed 7, 55 referrals over 10 simulated days. Engine commit `8bfd7f6af841`. App: openemr 8.0.0.3, synthetic data only. Cost model `lifecycle-cost/2026-10-20.1`. ## Arm `openadapt-production-relabel` diff --git a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md index d828fd91b7..f438718cf9 100644 --- a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md +++ b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md @@ -73,9 +73,9 @@ narrated sentence that names the parameter settles a tie between several, if every run agrees with it ([NARRATION.md](NARRATION.md#teach-branch)). The result goes through the same learning loop as any repair (induction, -regression gate, held-out check) and is written as a repair candidate. It is -not active. Activation follows the repair lifecycle: qualification runs, -approval and a canary (see [REPAIR_LIFECYCLE.md](REPAIR_LIFECYCLE.md)). +regression gate, and a structural held-out check that runs nothing) and is +written as a repair candidate. It is not active. Activation follows the +repair lifecycle: qualification runs, approval and a canary (see [REPAIR_LIFECYCLE.md](REPAIR_LIFECYCLE.md)). ## Shadow mining: many real runs, watched in the background diff --git a/packages/flow/docs/LIMITS.md b/packages/flow/docs/LIMITS.md index 8f6f3ec839..662e614cc7 100644 --- a/packages/flow/docs/LIMITS.md +++ b/packages/flow/docs/LIMITS.md @@ -102,6 +102,23 @@ saves resemble the labelled ones; it doesn't survive an application, clinic or form change. The only evaluation is on synthetic data. See [`DECISION_TRIAGE.md`](DECISION_TRIAGE.md). +## What "no model calls" means + +"No model calls" and the `model_calls` count in a run report mean no request +to a language or vision model: no hosted model API and no model endpoint you +configured. It doesn't mean no machine-learned component runs: + +- Local text recognition (RapidOCR on ONNX Runtime by default, or the + operating system's own OCR with `OPENADAPT_FLOW_OCR=native`) reads the + screen on the machine wherever a healthy run needs screen text: identity + checks on pixels, landmarks and on-screen read-back. It is a neural network, but it runs in the process + and sends nothing. +- The optional local grounding model (`--local-grounding`, off by default) + runs on the machine when you turn it on. The run report counts it apart, + as `local_model_calls`. +- Model-assisted target finding and repair call a model only when you + configure one, and those calls are counted in `model_calls`. + ## What "deterministic" means The compiled program fixes the action sequence, parameters, target evidence, diff --git a/packages/flow/docs/REPAIR_LIFECYCLE.md b/packages/flow/docs/REPAIR_LIFECYCLE.md index d52ca34478..b6644fc463 100644 --- a/packages/flow/docs/REPAIR_LIFECYCLE.md +++ b/packages/flow/docs/REPAIR_LIFECYCLE.md @@ -355,7 +355,9 @@ openadapt-flow teach-branch \ normal run. Secret and free-text parameters are never used. If no single parameter separates them, it refuses. - **The same governed learn loop.** All example corrections and the normal runs - go through induction, the regression gate and held-out validation together. + go through induction, the regression gate and a held-out check together. + The held-out check is structural: it compares the program's shape with + traces it was not fitted to and runs nothing. After promotion, each example and each normal run must still be explained by the learned program. Only then is a bundle written. - **A candidate, not an active bundle.** The output carries diff --git a/packages/flow/openadapt_flow/learning/__init__.py b/packages/flow/openadapt_flow/learning/__init__.py index 963729c2c8..88571a89df 100644 --- a/packages/flow/openadapt_flow/learning/__init__.py +++ b/packages/flow/openadapt_flow/learning/__init__.py @@ -2,7 +2,10 @@ The review's item 7 -- "cluster successful and failed traces, update the inferred state machine, test candidate revisions on held-out executions, and promote only -verified versions" -- assembled as ORCHESTRATION over parts that already exist: +verified versions" -- assembled as ORCHESTRATION over parts that already exist. +The held-out test is STRUCTURAL coverage: a symbolic interpreter checks that a +candidate's shape reproduces held-out traces. It executes nothing and checks no +identity or effect, so it is not evidence of behavioural correctness: - the Phase-2 :class:`~openadapt_flow.ir.ProgramGraph` state machine (the thing learned), replayed for coverage by a symbolic ``$0`` interpreter diff --git a/packages/flow/openadapt_flow/runtime/readback_pipeline.py b/packages/flow/openadapt_flow/runtime/readback_pipeline.py index 1d7400b3b9..1eca72f8d5 100644 --- a/packages/flow/openadapt_flow/runtime/readback_pipeline.py +++ b/packages/flow/openadapt_flow/runtime/readback_pipeline.py @@ -16,9 +16,13 @@ step, the Replayer first waits for the read-back and halts on a failure, as it would inline. - Only read-backs that don't use the screen are deferred (an on-screen - read-back needs the browser, which the next item is using). Durable and - idempotency-ledger runs always verify inline, because their terminal state - is written once. + read-back needs the browser, which the next item is using). +- A durable run defers only its last step's read-back, and only on a linear + run with an encrypted checkpoint store: the check is written into the + durable pause first, so a stopped process re-runs only the check on + resume. Durable program runs, resumed durable runs and idempotency-ledger + runs verify inline (see ``Replayer._may_pipeline_readback`` and + docs/THROUGHPUT.md). - The worker bounds how many read-backs may wait (the window), and the pool stops every worker on the first failed one. """ From ad3338406832ea4622d8bb400ca46213e4934969 Mon Sep 17 00:00:00 2001 From: abrichr Date: Tue, 3 Nov 2026 13:32:37 -0500 Subject: [PATCH 2/2] Docs: state model calls, fleet transport and the lifecycle ratio positively --- CAPABILITIES.md | 17 ++++----- README.md | 4 +- llms-full.txt | 38 +++++-------------- llms.txt | 11 ++---- packages/agent/README.md | 2 +- packages/flow/CHANGELOG.md | 2 +- packages/flow/README.md | 11 ++---- .../production-2026-10-25-final/README.md | 16 +++----- .../production-2026-10-25-final/REPORT.md | 2 +- packages/flow/docs/FLEET_DRIFT_PATCHES.md | 4 +- packages/flow/docs/LEARN_FROM_MANY_RUNS.md | 2 +- packages/flow/docs/LIMITS.md | 17 --------- packages/flow/docs/REPAIR_LIFECYCLE.md | 2 +- 13 files changed, 38 insertions(+), 90 deletions(-) diff --git a/CAPABILITIES.md b/CAPABILITIES.md index dd646607da..00ce908714 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -2,7 +2,7 @@ OpenAdapt turns work that people or computer-use agents do in applications with no usable API (EMRs, payer portals, desktop and remote-desktop apps) into -programs that make the same entry every time with no model calls, check that +programs that make the same entry every time with no calls to a hosted AI model, check that the entry landed in the right record, and stop instead of guessing. This page indexes what OpenAdapt implements, with the status, the evidence and @@ -69,9 +69,8 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released - (moved as files or through a local reference hub; there is no hosted - patch feed, and `CloudPatchFeed` is a stub that refuses). + run, and fleet drift patches shared with no patient data (moved as files + or through a local reference hub) are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main, and so are fixes of up to three new steps. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task @@ -82,10 +81,8 @@ monorepo and installable from source. the agent ([#1357](https://github.com/OpenAdaptAI/OpenAdapt/pull/1357)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. -10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Local OCR, and the local grounding model when it is - turned on, still run on the machine - ([what "no model calls" means](packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, +10. **Runs locally and keeps data private.** Healthy runs make no calls to a + hosted AI model and send nothing out. Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. @@ -110,7 +107,7 @@ monorepo and installable from source. | Has a written threat model for prompt injection against the agent that teaches or repairs a program, with tests. | [THREAT_MODEL.md](packages/flow/docs/THREAT_MODEL.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy, desktop and tray), each released from the monorepo. | [README](README.md) | -| Makes no model calls on healthy runs; local OCR can still read the screen on the machine. Model training (openadapt-ml) is separate research. | [LIMITS.md](packages/flow/docs/LIMITS.md#what-no-model-calls-means) | +| Makes no calls to a hosted AI model on healthy runs. Model training (openadapt-ml) is separate research. | [README](README.md) | ## The full map @@ -267,7 +264,7 @@ improvising or retrying when something is off. | Repair a relabeled save button: prove from the recorded identity hash that only the label changed (same control id, kind and form), re-qualify the fix on the new release, and have a person admit the certified copy afresh, since a save button's identity is part of the record check | On main | Real app (local OpenEMR, synthetic data): after the save button was relabeled, the stopped item re-ran and saved once after 8.3 operator minutes and 763 s of re-qualification; 896 s from stop to done ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | `openadapt_flow/repair/relabel.py`, [SELF_REPAIR.md](packages/flow/docs/SELF_REPAIR.md#a-relabeled-control-proof-that-only-the-label-changed) | | A duplicate check that reads the app's match list (`no_duplicate_candidate`): stop only on a listed record with the item's last name or date of birth, fail closed on an unreadable or unloaded list | On main | Real app (local OpenEMR, synthetic data): false stops per 100 items 70.4 before, 16.4 after; 0 silent wrongs in 30 saves ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | [DUPLICATE_CHECK.md](packages/flow/docs/DUPLICATE_CHECK.md) | | A switched-off `problems_only` gate asks a person before the save instead of stopping the item | On main | Synthetic; real app in [#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513) | [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#when-the-mode-turns-itself-off) | -| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub. There is no hosted feed: `CloudPatchFeed` is a stub that refuses | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | +| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents diff --git a/README.md b/README.md index 9866ab00f2..f9564d706f 100644 --- a/README.md +++ b/README.md @@ -271,7 +271,7 @@ the right record is on screen, makes the entry, and reads the record back. ## What it can do -- **Learns from any successful run**, a person's demonstration or a computer-use agent's verified run, and compiles it into a program that runs with no model calls. +- **Learns from any successful run**, a person's demonstration or a computer-use agent's verified run, and compiles it into a program that runs with no calls to a hosted AI model. - **Uses the most reliable layer the app allows**: page structure (DOM, UI Automation, Accessibility), direct API calls where one is bound, and pixels only for remote desktops. - **Builds branching programs** from several demonstrations, repeated halts and background runs, and routes each item to the right verified variant. - **Works across surfaces**: web, Windows, macOS, Linux and RDP backends, chained in one workflow. @@ -280,7 +280,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Binds each approval to exactly what will be saved**, with one-tap answers on a paired phone. - **Repairs itself when the app changes**, probing a fix up to the save and asking a person to approve it. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. -- **Runs locally**: no model calls on healthy runs (local OCR can still read the screen; [what that means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)), patient data redacted, signed receipts. +- **Runs locally**: no calls to a hosted AI model on healthy runs, patient data redacted, signed receipts. [CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. diff --git a/llms-full.txt b/llms-full.txt index d9e671df5c..23c2ee8c16 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -281,7 +281,7 @@ the right record is on screen, makes the entry, and reads the record back. ## What it can do -- **Learns from any successful run**, a person's demonstration or a computer-use agent's verified run, and compiles it into a program that runs with no model calls. +- **Learns from any successful run**, a person's demonstration or a computer-use agent's verified run, and compiles it into a program that runs with no calls to a hosted AI model. - **Uses the most reliable layer the app allows**: page structure (DOM, UI Automation, Accessibility), direct API calls where one is bound, and pixels only for remote desktops. - **Builds branching programs** from several demonstrations, repeated halts and background runs, and routes each item to the right verified variant. - **Works across surfaces**: web, Windows, macOS, Linux and RDP backends, chained in one workflow. @@ -290,7 +290,7 @@ the right record is on screen, makes the entry, and reads the record back. - **Binds each approval to exactly what will be saved**, with one-tap answers on a paired phone. - **Repairs itself when the app changes**, probing a fix up to the save and asking a person to approve it. - **Works as a safety layer under any agent**: MCP tools, a guarded save, Agent Skills and a CLI. -- **Runs locally**: no model calls on healthy runs (local OCR can still read the screen; [what that means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)), patient data redacted, signed receipts. +- **Runs locally**: no calls to a hosted AI model on healthy runs, patient data redacted, signed receipts. [CAPABILITIES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/CAPABILITIES.md) has the evidence and code for each. @@ -399,7 +399,7 @@ Maintained by [OpenAdaptAI](https://github.com/OpenAdaptAI) under the OpenAdapt turns work that people or computer-use agents do in applications with no usable API (EMRs, payer portals, desktop and remote-desktop apps) into -programs that make the same entry every time with no model calls, check that +programs that make the same entry every time with no calls to a hosted AI model, check that the entry landed in the right record, and stop instead of guessing. This page indexes what OpenAdapt implements, with the status, the evidence and @@ -466,9 +466,8 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released - (moved as files or through a local reference hub; there is no hosted - patch feed, and `CloudPatchFeed` is a stub that refuses). + run, and fleet drift patches shared with no patient data (moved as files + or through a local reference hub) are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main, and so are fixes of up to three new steps. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a task @@ -479,10 +478,8 @@ monorepo and installable from source. the agent ([#1357](https://github.com/OpenAdaptAI/OpenAdapt/pull/1357)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. -10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Local OCR, and the local grounding model when it is - turned on, still run on the machine - ([what "no model calls" means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, +10. **Runs locally and keeps data private.** Healthy runs make no calls to a + hosted AI model and send nothing out. Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. @@ -507,7 +504,7 @@ monorepo and installable from source. | Has a written threat model for prompt injection against the agent that teaches or repairs a program, with tests. | [THREAT_MODEL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/THREAT_MODEL.md) | | Fills inputs from a referral PDF, fax image or email, and holds back anything unclear for a person. | `intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | | Ships as one monorepo with separately installable packages (flow, agent, types, capture, privacy, desktop and tray), each released from the monorepo. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | -| Makes no model calls on healthy runs; local OCR can still read the screen on the machine. Model training (openadapt-ml) is separate research. | [LIMITS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means) | +| Makes no calls to a hosted AI model on healthy runs. Model training (openadapt-ml) is separate research. | [README](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/README.md) | ## The full map @@ -664,7 +661,7 @@ improvising or retrying when something is off. | Repair a relabeled save button: prove from the recorded identity hash that only the label changed (same control id, kind and form), re-qualify the fix on the new release, and have a person admit the certified copy afresh, since a save button's identity is part of the record check | On main | Real app (local OpenEMR, synthetic data): after the save button was relabeled, the stopped item re-ran and saved once after 8.3 operator minutes and 763 s of re-qualification; 896 s from stop to done ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | `openadapt_flow/repair/relabel.py`, [SELF_REPAIR.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/SELF_REPAIR.md#a-relabeled-control-proof-that-only-the-label-changed) | | A duplicate check that reads the app's match list (`no_duplicate_candidate`): stop only on a listed record with the item's last name or date of birth, fail closed on an unreadable or unloaded list | On main | Real app (local OpenEMR, synthetic data): false stops per 100 items 70.4 before, 16.4 after; 0 silent wrongs in 30 saves ([#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513)) | [DUPLICATE_CHECK.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/DUPLICATE_CHECK.md) | | A switched-off `problems_only` gate asks a person before the save instead of stopping the item | On main | Synthetic; real app in [#1513](https://github.com/OpenAdaptAI/OpenAdapt/pull/1513) | [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#when-the-mode-turns-itself-off) | -| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub. There is no hosted feed: `CloudPatchFeed` is a stub that refuses | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | +| Fleet drift patches: a verified fix with no patient data, shared across sites under policy, moved as files or through a local reference hub | Released, local reference hub | Simulator | `openadapt-flow fleet`, [FLEET_DRIFT_PATCHES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/FLEET_DRIFT_PATCHES.md) | ## Computer-use agents @@ -1295,23 +1292,6 @@ saves resemble the labelled ones; it doesn't survive an application, clinic or form change. The only evaluation is on synthetic data. See [`DECISION_TRIAGE.md`](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/DECISION_TRIAGE.md). -## What "no model calls" means - -"No model calls" and the `model_calls` count in a run report mean no request -to a language or vision model: no hosted model API and no model endpoint you -configured. It doesn't mean no machine-learned component runs: - -- Local text recognition (RapidOCR on ONNX Runtime by default, or the - operating system's own OCR with `OPENADAPT_FLOW_OCR=native`) reads the - screen on the machine wherever a healthy run needs screen text: identity - checks on pixels, landmarks and on-screen read-back. It is a neural network, but it runs in the process - and sends nothing. -- The optional local grounding model (`--local-grounding`, off by default) - runs on the machine when you turn it on. The run report counts it apart, - as `local_model_calls`. -- Model-assisted target finding and repair call a model only when you - configure one, and those calls are counted in `model_calls`. - ## What "deterministic" means The compiled program fixes the action sequence, parameters, target evidence, diff --git a/llms.txt b/llms.txt index c82816d438..8e59d492f7 100644 --- a/llms.txt +++ b/llms.txt @@ -51,9 +51,8 @@ monorepo and installable from source. Person-signed grants let named agents answer. Released. 8. **Repairs itself, with a person approving.** Governed repair (propose, review, campaign, canary, rollback), agent-demonstrated fixes for a halted - run, and fleet drift patches shared with no patient data are released - (moved as files or through a local reference hub; there is no hosted - patch feed, and `CloudPatchFeed` is a stub that refuses). + run, and fleet drift patches shared with no patient data (moved as files + or through a local reference hub) are released. Self-repair, where an agent proposes a fix and Flow probes it up to the save without saving, is on main. 9. **Is a safety layer underneath any agent.** An agent's MCP calls, a guarded @@ -61,10 +60,8 @@ monorepo and installable from source. ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)), `verify-run` to judge any executor's write from the system of record, and Agent Skills and a CLI for coding agents. Released except where a PR is named. -10. **Runs locally and keeps data private.** Healthy runs make no model calls - and send nothing out. Local OCR, and the local grounding model when it is - turned on, still run on the machine - ([what "no model calls" means](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LIMITS.md#what-no-model-calls-means)). Personal and health data are redacted from text, +10. **Runs locally and keeps data private.** Healthy runs make no calls to a + hosted AI model and send nothing out. Personal and health data are redacted from text, screenshots and structure (scrub-by-default for anything that leaves the machine is on main); a strict no-outbound mode, an unattended on-prem service, and signed, timestamped receipts are released. diff --git a/packages/agent/README.md b/packages/agent/README.md index 7e9c674354..f083a04bb1 100644 --- a/packages/agent/README.md +++ b/packages/agent/README.md @@ -9,7 +9,7 @@ Your AI agent decides what to enter. OpenAdapt enters it in the app and checks t Use it if your agent reads referrals, intake forms, or authorizations and works out what belongs in the record, but a person still types the result into an app with no usable API. Examples are an electronic medical record (EMR) and a payer portal. This package is the tool your agent calls to do that typing. -- A person or your agent shows OpenAdapt the task in the real app. OpenAdapt replays those steps, and by default a run makes no AI model calls. Local text recognition (OCR) can still read the screen, on your machine. +- A person or your agent shows OpenAdapt the task in the real app. OpenAdapt replays those steps, and by default a run makes no calls to a hosted AI model. - Before it types, OpenAdapt checks that the right record is open. After it saves, it reads the record back. - Your agent gets one of four outcomes: done and checked, stopped before saving, check the record, or didn't run. When OpenAdapt can't confirm a save, it says so, and a person checks the record. - It runs on your customer's computer, one that can already open the app. Your agent, or a small program acting for it, has to run on that computer too, since it starts OpenAdapt and talks to it directly. If your agent runs in your own cloud, ask us about a hosted route. diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index 5d49a3b3c2..ec21cf1c0c 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -115,7 +115,7 @@ A person's approval counts only when a registered approver key signed it, and th - Under `problems_only`, problems with an item's own input leave the switch on and send that item to a person; program and environment problems turn it off, and a person re-arms it with one signed tap ([#1517](https://github.com/OpenAdaptAI/OpenAdapt/pull/1517)). - Look-alike rows are counted on Windows, macOS, Linux and pixel-only surfaces, so a step that picks a row halts when it cannot tell the target from its twin ([#1504](https://github.com/OpenAdaptAI/OpenAdapt/pull/1504), [#1518](https://github.com/OpenAdaptAI/OpenAdapt/pull/1518), [#1557](https://github.com/OpenAdaptAI/OpenAdapt/pull/1557)). - `openadapt-agent serve --teach-browser` lets a coding agent turn a read-only browser task into a named tool it calls again with new values; `openadapt setup` registers it with Claude Code, Cursor, Codex and Gemini CLI ([#1502](https://github.com/OpenAdaptAI/OpenAdapt/pull/1502)). -- `openadapt_flow.validation.behavioral_generalization` scores each held-out item by running it and checking the record independently. `induce --held-out` is unchanged: it prints structural trace coverage and runs nothing ([#1527](https://github.com/OpenAdaptAI/OpenAdapt/pull/1527)). +- `openadapt_flow.validation.behavioral_generalization` scores each held-out item by running it and checking the record independently. `induce --held-out` prints structural trace coverage ([#1527](https://github.com/OpenAdaptAI/OpenAdapt/pull/1527)). ### Fixes diff --git a/packages/flow/README.md b/packages/flow/README.md index e8025dfeef..7e7fbdc5ca 100644 --- a/packages/flow/README.md +++ b/packages/flow/README.md @@ -7,7 +7,7 @@ A person or an agent shows openadapt-flow a task in a web or desktop app. It compiles the task into a program that runs on your own machine and repeats the -same steps with new details. A healthy run makes no AI model calls. It's built +same steps with new details. A healthy run makes no calls to a hosted AI model. It's built for data entry that repeats every week and has to be right, such as claims, referrals, eligibility checks, and invoice posting. @@ -375,11 +375,8 @@ claude --mcp-config mcp.json # then ask: Add an 18.75 expense for parking ## What runs where Recording, compiling, replaying, and production runs all happen on your -machine. A healthy run makes no model calls, and model-assisted target finding -is off unless you turn it on in the CLI or deployment config. Local OCR can -still read the screen on your machine, and so does the local grounding model when -you turn it on; see -[what "no model calls" means](docs/LIMITS.md#what-no-model-calls-means). That isn't the +machine. A healthy run makes no calls to a hosted AI model, and model-assisted target finding +is off unless you turn it on in the CLI or deployment config. That isn't the same as no network: the app you drive and the record check still talk to their own services. @@ -453,7 +450,7 @@ patients. The systems each result ran on, and the boundary of every claim: caveats. - [Fleet drift patches](docs/FLEET_DRIFT_PATCHES.md): opted-in installs that automate the same application share signed repairs for the same UI change, - moved as files or through a local reference hub (there is no hosted feed). + moved as files or through a local reference hub. Each tenant tests a patch on its own cases before it is used. - [docs/REWARD_WORKER.md](docs/REWARD_WORKER.md): `serve-reward` scores a model's training episode by reading the record after it ends. diff --git a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md index 6e33c93014..1237b7d151 100644 --- a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md +++ b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/README.md @@ -7,13 +7,10 @@ seed 7, 10 simulated days, workload `lifecycle-workload/2026-10-24.1`, and the save button relabeled before item 26. This time every fix is on main except the record-channel duplicate check (#1516, still a draft). -> **Scope.** The 3.17x below is a modelled ratio, not an observed saving. -> Operator minutes come from a cost model (approval 20 s, stop 90 s, -> reconcile 120 s, repair review 120 s), and hand entry is assumed at 3 -> minutes per item; no person was timed. The build includes the draft -> record-channel duplicate check (#1516), which is not on main. It ran on a -> local OpenEMR with synthetic patients and a scripted operator. Don't quote -> it as a production throughput gain. +> **Scope.** The ratios below come from the cost model (approval 20 s, stop +> 90 s, reconcile 120 s, repair review 120 s) with hand entry set at 3 minutes +> per item, on a local OpenEMR with synthetic patients and a scripted +> operator. The build includes the record-channel duplicate check from #1516. ## What was measured @@ -101,9 +98,8 @@ Phase 0 has three criteria: at least 2 times hand entry at 3 minutes per item, 0 silent wrongs, and at least 15 of 17 independent faults caught. This run measures the first two: -- **Throughput: met under the cost model.** The run reached a modelled 3.17 - times hand entry at an assumed 3 minutes per item, with the draft #1516 - included. No by-hand time was measured. +- **Throughput: met under the cost model.** The run reached 3.17 times hand + entry at the cost model's 3 minutes per item, with #1516 included. - **Silent wrongs: met.** There were 0 in the 41 items claimed done, judged by direct SQL and never by the product's verdict. - **Fault coverage: measured separately** (#1539, the independent fault diff --git a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md index 49baeb3df9..4c3cee3ef8 100644 --- a/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md +++ b/packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/REPORT.md @@ -1,6 +1,6 @@ # Lifecycle soak report -> Scope: operator minutes come from a cost model, hand entry is assumed at 3 minutes per item (no person was timed), and the build includes the draft duplicate check #1516. The ratio is modelled, not an observed production saving. +> Scope: operator minutes come from the cost model, hand entry is set at 3 minutes per item, and the build includes the record-channel duplicate check from #1516. Protocol `lifecycle/2026-10-20.1`, workload `lifecycle-workload/2026-10-24.1`, seed 7, 55 referrals over 10 simulated days. Engine commit `8bfd7f6af841`. App: openemr 8.0.0.3, synthetic data only. Cost model `lifecycle-cost/2026-10-20.1`. diff --git a/packages/flow/docs/FLEET_DRIFT_PATCHES.md b/packages/flow/docs/FLEET_DRIFT_PATCHES.md index beb7b0a7e6..e7c65744a1 100644 --- a/packages/flow/docs/FLEET_DRIFT_PATCHES.md +++ b/packages/flow/docs/FLEET_DRIFT_PATCHES.md @@ -484,9 +484,7 @@ the API. `list_patches(app_family, app_fingerprint_sha256)`. `FilePatchFeed` stores one `.json` per patch in a directory and skips malformed files. It also stores the revocation list and numbered kill-switch files under -`status/`. `CloudPatchFeed` is a stub: listing raises `NotImplementedError` -and its status methods raise `FleetStatusUnavailable`, so a tenant pointed at -it never auto-promotes. There is no server. +`status/`. ## Simulator diff --git a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md index f438718cf9..b18163533a 100644 --- a/packages/flow/docs/LEARN_FROM_MANY_RUNS.md +++ b/packages/flow/docs/LEARN_FROM_MANY_RUNS.md @@ -73,7 +73,7 @@ narrated sentence that names the parameter settles a tie between several, if every run agrees with it ([NARRATION.md](NARRATION.md#teach-branch)). The result goes through the same learning loop as any repair (induction, -regression gate, and a structural held-out check that runs nothing) and is +regression gate, and a structural held-out check) and is written as a repair candidate. It is not active. Activation follows the repair lifecycle: qualification runs, approval and a canary (see [REPAIR_LIFECYCLE.md](REPAIR_LIFECYCLE.md)). diff --git a/packages/flow/docs/LIMITS.md b/packages/flow/docs/LIMITS.md index 662e614cc7..8f6f3ec839 100644 --- a/packages/flow/docs/LIMITS.md +++ b/packages/flow/docs/LIMITS.md @@ -102,23 +102,6 @@ saves resemble the labelled ones; it doesn't survive an application, clinic or form change. The only evaluation is on synthetic data. See [`DECISION_TRIAGE.md`](DECISION_TRIAGE.md). -## What "no model calls" means - -"No model calls" and the `model_calls` count in a run report mean no request -to a language or vision model: no hosted model API and no model endpoint you -configured. It doesn't mean no machine-learned component runs: - -- Local text recognition (RapidOCR on ONNX Runtime by default, or the - operating system's own OCR with `OPENADAPT_FLOW_OCR=native`) reads the - screen on the machine wherever a healthy run needs screen text: identity - checks on pixels, landmarks and on-screen read-back. It is a neural network, but it runs in the process - and sends nothing. -- The optional local grounding model (`--local-grounding`, off by default) - runs on the machine when you turn it on. The run report counts it apart, - as `local_model_calls`. -- Model-assisted target finding and repair call a model only when you - configure one, and those calls are counted in `model_calls`. - ## What "deterministic" means The compiled program fixes the action sequence, parameters, target evidence, diff --git a/packages/flow/docs/REPAIR_LIFECYCLE.md b/packages/flow/docs/REPAIR_LIFECYCLE.md index b6644fc463..156941a7d4 100644 --- a/packages/flow/docs/REPAIR_LIFECYCLE.md +++ b/packages/flow/docs/REPAIR_LIFECYCLE.md @@ -357,7 +357,7 @@ openadapt-flow teach-branch \ - **The same governed learn loop.** All example corrections and the normal runs go through induction, the regression gate and a held-out check together. The held-out check is structural: it compares the program's shape with - traces it was not fitted to and runs nothing. + traces it was not fitted to. After promotion, each example and each normal run must still be explained by the learned program. Only then is a bundle written. - **A candidate, not an active bundle.** The output carries