diff --git a/CAPABILITIES.md b/CAPABILITIES.md index 33c5ec13e..231cb0442 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -180,7 +180,7 @@ improvising or retrying when something is off. | Review stops: every stop as one card with the evidence side by side (the item next to each possible match, field by field), answered with one key or tap on the computer or a paired phone, signed with the approver key, with batches for similar stops. The queue presents and collects; a changed or expired card changes nothing | Released ([#1519](https://github.com/OpenAdaptAI/OpenAdapt/pull/1519)) | Unit tests; a keystroke-level estimate on 20 soak stops (26.7 s to 8.2 s to decide, a model, not a person) | `packages/desktop/openadapt_desktop/stop_queue.py`, [benchmark/stop_queue](packages/desktop/benchmark/stop_queue/README.md) | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | Released ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| When the demonstration never reads the saved record back, find the read by probing the collection and item reads the captured write implies (GET only, local copy of the app only, every probe in the proof) and use one only when it returns the written values and no write outside it could pass unseen; otherwise refuse or leave the read unavailable for a person. Also sees a save with an unfamiliar button label from its POST, and marks a generated REST read complete when it was declared or seen whole and unpaged | On main | Synthetic (local test app; zero-config benchmark on MockMed and MockLoan) | `openadapt-flow scaffold-verifier --probe-reads`, `openadapt_flow/read_discovery.py` | +| When the demonstration never reads the saved record back, find the read by probing the collection and item reads the captured write implies (GET only, local copy of the app only, every probe in the proof) and use one only when it returns the written values and no write outside it could pass unseen. Before refusing a collection read, look for a prefix root that holds every record of the collection and the siblings the app read, and read the whole store from it; otherwise refuse or leave the read unavailable for a person. Also sees a save with an unfamiliar button label from its POST, and marks a generated REST read complete when it was declared or seen whole and unpaged | On main | Synthetic (local test app; zero-config benchmark on MockMed and MockLoan) | `openadapt-flow scaffold-verifier --probe-reads`, `openadapt_flow/read_discovery.py` | | Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | Released ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | | Learn the record check from the recording, when the person looks the record up after saving | Released ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | | A record check's duplicate key can be several fields together, such as last name, first name and birth date: the check fails when more than one record carries all of them, counting records that were already there | Released ([#1393](https://github.com/OpenAdaptAI/OpenAdapt/issues/1393)) | Synthetic (OpenEMR-shaped patient rows) | verify.yaml `record.duplicate_key: [lname, fname, DOB]`, `confirm_params(idempotency_key=[...])`, `Effect.composite_key` | diff --git a/llms-full.txt b/llms-full.txt index f0ca0ced2..e188d0f26 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -614,7 +614,7 @@ improvising or retrying when something is off. | Review stops: every stop as one card with the evidence side by side (the item next to each possible match, field by field), answered with one key or tap on the computer or a paired phone, signed with the approver key, with batches for similar stops. The queue presents and collects; a changed or expired card changes nothing | Released ([#1519](https://github.com/OpenAdaptAI/OpenAdapt/pull/1519)) | Unit tests; a keystroke-level estimate on 20 soak stops (26.7 s to 8.2 s to decide, a model, not a person) | `packages/desktop/openadapt_desktop/stop_queue.py`, [benchmark/stop_queue](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/desktop/benchmark/stop_queue/README.md) | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | | Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | Released ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` | -| When the demonstration never reads the saved record back, find the read by probing the collection and item reads the captured write implies (GET only, local copy of the app only, every probe in the proof) and use one only when it returns the written values and no write outside it could pass unseen; otherwise refuse or leave the read unavailable for a person. Also sees a save with an unfamiliar button label from its POST, and marks a generated REST read complete when it was declared or seen whole and unpaged | On main | Synthetic (local test app; zero-config benchmark on MockMed and MockLoan) | `openadapt-flow scaffold-verifier --probe-reads`, `openadapt_flow/read_discovery.py` | +| When the demonstration never reads the saved record back, find the read by probing the collection and item reads the captured write implies (GET only, local copy of the app only, every probe in the proof) and use one only when it returns the written values and no write outside it could pass unseen. Before refusing a collection read, look for a prefix root that holds every record of the collection and the siblings the app read, and read the whole store from it; otherwise refuse or leave the read unavailable for a person. Also sees a save with an unfamiliar button label from its POST, and marks a generated REST read complete when it was declared or seen whole and unpaged | On main | Synthetic (local test app; zero-config benchmark on MockMed and MockLoan) | `openadapt-flow scaffold-verifier --probe-reads`, `openadapt_flow/read_discovery.py` | | Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | Released ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` | | Learn the record check from the recording, when the person looks the record up after saving | Released ([#1290](https://github.com/OpenAdaptAI/OpenAdapt/pull/1290)) | Synthetic | `record --verify` | | A record check's duplicate key can be several fields together, such as last name, first name and birth date: the check fails when more than one record carries all of them, counting records that were already there | Released ([#1393](https://github.com/OpenAdaptAI/OpenAdapt/issues/1393)) | Synthetic (OpenEMR-shaped patient rows) | verify.yaml `record.duplicate_key: [lname, fname, DOB]`, `confirm_params(idempotency_key=[...])`, `Effect.composite_key` | diff --git a/packages/flow/CHANGELOG.md b/packages/flow/CHANGELOG.md index bb68a6a4c..2f914ac51 100644 --- a/packages/flow/CHANGELOG.md +++ b/packages/flow/CHANGELOG.md @@ -20,6 +20,7 @@ ### Changed - `scaffold-verifier` closes three gaps the zero-config benchmark found. A generated REST read now carries `records_complete: true` when `--complete-read` declares it or one unpaged response was seen to hold the whole collection; otherwise it stays incomplete and `verify-run` returns INDETERMINATE. A step counts as a write when the capture shows the page sending a POST, PUT, PATCH or DELETE to its own origin, or its store snapshots differ, not only when its label looks like Save. `--probe-reads` (with optional `--probe-base`) finds a read for an app that never reads the saved record back: it probes the collection and item reads the write implies with GET on the local machine, uses one only when it returns the written values and nothing outside it could change unseen (otherwise it refuses and writes the TODO draft), and records every probe (`read_discovery` in the proof, or `read_discovery.json` when none confirms). +- `scaffold-verifier --probe-reads` now looks for a read that covers the whole save before it refuses a probed collection read. It probes the sibling collections under the same prefix that the app itself read (path only, from its captured GETs) and the prefix root. It uses the root instead of the collection only when the root holds the written record and every record the collection and siblings returned, in one unpaged response, every captured write went under the prefix, and every top-level value is a list of objects or a scalar (an object, such as a nested collection or an id-keyed map, refuses the root). The check reads every list the root returns, an empty one included, and the save may add exactly one row to the list holding the written record. It also reads any list that appears later (`records_other_lists`), where the save may add no row; a list it was built on that disappears makes the read unreadable (`records_list_keys`). Otherwise the refusal stands, and its reason lists what was probed. Probes still send only GET, only to the local machine, with no credentials, proxy or redirects. `read_discovery.json` is now also written when a probed read is used. ## v2.7.0 (2026-11-09) diff --git a/packages/flow/benchmark/zero_config_verify/PROTOCOL.md b/packages/flow/benchmark/zero_config_verify/PROTOCOL.md index 3d92444d4..1ee917836 100644 --- a/packages/flow/benchmark/zero_config_verify/PROTOCOL.md +++ b/packages/flow/benchmark/zero_config_verify/PROTOCOL.md @@ -226,3 +226,54 @@ A read observed in the recording's own traffic (`Z-net-read`, check. It runs under this protocol once the flags are deleted. That case is left to a follow-up issue. The third run's numbers are unchanged by this note. + +### 2026-11-09, before the fourth run (look for a read that covers the save) + +The third run refused MockLoan's probed read (`/api/disbursements`) because a +write outside it would pass. Issue #2131 changes `--probe-reads` to look for +a read that covers the whole save before it refuses. The candidates are the +sibling collections under the same prefix that the app itself read (from its +own captured `GET` requests, path only) and the prefix root (`GET ` +and `GET /`). The probe rules are unchanged: `GET` only, the local +machine only, no credentials, no redirects, and every probe recorded in +`read_discovery.json` and the proof. The root replaces the collection read +only when it holds the written record, holds every record the collection and +each sibling returned, is one unpaged response, and every captured write went +under the prefix. The check then reads every list the root returns, and the +save may add exactly one row to it. A read that still leaves an open +`collateral_unaudited` miss is refused, as before, and the reason now says +what the search probed. + +The change is app-generic (`scripts/check_engine_generality.py` passes), was +tested against separate local test apps, and adds no candidate names: the +generator does not guess store-wide routes. A generation-only smoke run +(writes nothing) on this commit found that MockLoan's app reads no other +collection under `/api`, and that `GET /api` and `GET /api/` return 404. So +the expected outcome is that `Z-net-probed` on MockLoan stays refused, with +that reason recorded. MockMed is out of scope (no JSON read route) and its +arm cannot change. + +The fourth run re-runs every arm on the same cases. The third run's data is +kept as `results-run3.json`. The thresholds, the baselines, the ground truth +and the trial cases are unchanged. + +### 2026-11-09, before the fourth run (review fix) + +Review of the change above found a false CONFIRMED: a root list that was empty +when probed (`"fees": []`), or a collection the root left out while empty, got +no read, so a stray row written there later passed. The fix, made before the +fourth run: the check reads every top-level list the root returns, empty ones +included; one more read takes every list under a key the probe did not see, +and the save may add no row there; a reviewed list that disappears makes the +read unreadable. The fault test injects a stray row into each of these. Nothing +else in this protocol changes, and the expected MockLoan outcome (refused, +`GET /api` returns 404) is the same. + +### 2026-11-09, before the fourth run (second review fix) + +The second review found two more shapes a root could hide records in: a +collection nested in an object, and an id-keyed map. Ruling: fail closed on +shape. The root is promoted only when every top-level value is a list of +objects or a scalar; otherwise the reason is "root holds a non-list value that +may contain records: ". At check time, an object value in the root makes +the read unreadable. The fourth run uses this commit. Nothing else changes. diff --git a/packages/flow/benchmark/zero_config_verify/RESULTS.md b/packages/flow/benchmark/zero_config_verify/RESULTS.md index d92b4b83b..31c8b99aa 100644 --- a/packages/flow/benchmark/zero_config_verify/RESULTS.md +++ b/packages/flow/benchmark/zero_config_verify/RESULTS.md @@ -1,17 +1,19 @@ # Results: generated verifiers vs hand-written verifiers under planted faults Protocol: [PROTOCOL.md](PROTOCOL.md). It was committed before any trial -(42415a612). Three deviations, each committed before the run it affects: +(42415a612). Each deviation was committed before the run it affects: - c4a3be060: the `hand-written+complete` baseline; - 79df8b96c: the second run, with the fixed generator and the new `Z-net-probed` arm; - bedb622b9: the third run, which refuses a probed read that can't see the - whole save. + whole save; +- a4e8ffdea, f484a9933 and ae4ecc75e: the fourth run, in which + `--probe-reads` looks for a read that covers the whole save before it + refuses (#2131), with two review fixes. -This file reports the **third run**, made on 2026-11-09 at commit cfa11472b. -That commit is this branch with #2129 (`feat/zero-config-read-discovery`, -32997eeac) merged in. The run had 69 trials (36 MockMed, 33 MockLoan), used +This file reports the **fourth run**, made on 2026-11-10 (UTC) at commit +ae4ecc75e (#2147). The run had 69 trials (36 MockMed, 33 MockLoan), used headless Chromium on localhost only, made no model calls and used synthetic data only. Raw data is in [results.json](results.json), and the generated files are in [generated/](generated/). Earlier runs are kept and summarized @@ -19,12 +21,14 @@ at the end: - [results-run1.json](results-run1.json): the generator as it was on `main`; - [results-run2.json](results-run2.json): the fixed generator, before the - refusal. + refusal; +- [results-run3.json](results-run3.json): the refusal, before the search for + a covering read. ## Verdict -**The protocol's rule is met on these fixtures only through an assisted arm -whose file the generator labels a draft.** +**Unchanged from the third run. The protocol's rule is met on these fixtures +only through an assisted arm whose file the generator labels a draft.** - `Z-net-read` passes all three kill thresholds on both apps. Its demonstration also loads the store's record list (`GET /api/db`) before @@ -37,33 +41,46 @@ whose file the generator labels a draft.** - The only unassisted pass is `Z-db` and `Z-db-gate` on MockLoan, whose store is a file the generator can read. -The new arm, `Z-net-probed`, uses only the app's own traffic. It has **no -runnable verifier on either app, and so no false accepts**: +`Z-net-probed` uses only the app's own traffic. It still has **no runnable +verifier on either app, and so no false accepts**: - **MockMed: read unavailable.** The app has no JSON read route for the collection it writes, so all 4 probes returned 404: - `GET /api/encounter` and `GET /api/encounters`; - `GET /api/encounter/1` and `GET /api/encounters/1`. - The generator wrote the TODO draft and no check. -- **MockLoan (held out): refused.** The probe found and confirmed - `GET /api/disbursements`, which returns the record with all four written - fields. The fault test then showed that a write outside that collection - would pass (`collateral_unaudited`). So the generator wrote no runnable - check, only the TODO draft. `read_discovery.json` and the proof record the - status as `refused`, with `needs_operator_confirmation: true` and the - reason. - - In the second run, before the refusal, this arm accepted all 3 - `collateral` runs. - -On MockLoan, the database arms (`Z-db`, `Z-db-gate`) also pass. They were -blocked before generation in the first run. - -Every arm that produced a verifier now returns a definite verdict: -0 INDETERMINATE in 69 trials. In the first run, every generated REST check -returned INDETERMINATE. The ground truth agreed with each fault mode's design -label on all 69 trials, and no verdict carried an error. + The generator wrote the TODO draft and no check. The search for a covering + read runs only after a read is confirmed, so it did not run here. +- **MockLoan (held out): refused, with the covering search recorded.** The + probe confirmed `GET /api/disbursements`, which returns the record with all + four written fields. The fault test showed that a write outside that + collection would pass (`collateral_unaudited`). The generator then looked + for a read that covers the whole save: + - the app read no other collection under `/api`, so there was no sibling + to probe; + - `GET /api` and `GET /api/` both returned 404. + + Nothing covered the save, so the refusal stands. `read_discovery.json` and + the proof record all 4 probes, the status `refused`, and this reason: + "No read covering the whole save was found: GET /api: no read here (status + 404); GET /api/: no read here (status 404); the app read no other + collection under /api." + + The store-wide read (`/api/db`) exists, but the app never requests it and + it is not under a name the generator could derive. The generator does not + guess route names. + +On MockLoan, the database arms (`Z-db`, `Z-db-gate`) also pass. + +Every arm that produced a verifier returns a definite verdict: 0 +INDETERMINATE in 69 trials. The ground truth agreed with each fault mode's +design label on all 69 trials, and no verdict carried an error. + +One baseline changed: the committed hand-written files now carry +`records_complete: true` (added by #1784 after the third run's commit). So +`hand-written` (as committed) now matches `hand-written+complete`: 0 false +rejects instead of 12 of 12. The thresholds still compare against +`hand-written+complete`, as the protocol says. ## Summary table @@ -74,7 +91,7 @@ label on all 69 trials, and no verdict carried an error. | App | Verifier | Class | FA / wrong runs | FR / correct runs | Kill check | |---|---|---|---|---|---| -| MockMed | `hand-written` (as committed) | baseline | 0 / 24 | 12 / 12 | (not used) | +| MockMed | `hand-written` (as committed) | baseline | 0 / 24 | 0 / 12 | (not used) | | MockMed | `hand-written+complete` | baseline | 0 / 24 | 0 / 12 | (threshold baseline) | | MockMed | `Z-net` | zero-config | no runnable verifier (TODO draft) | | not ready | | MockMed | `Z-net-probed` | zero-config | no runnable verifier (read unavailable: no read route) | | not ready | @@ -83,10 +100,10 @@ label on all 69 trials, and no verdict carried an error. | MockMed | `D-net-read-complete` | declared | 0 / 24 | 0 / 12 | ready | | MockMed | `P-record-verify` | read path given | 0 / 24 | 0 / 12 | ready | | MockMed | `P-scaffold-verifier` | read path given | 0 / 24 | 0 / 12 | ready | -| MockLoan | `hand-written` (as committed) | baseline | 0 / 21 | 12 / 12 | (not used) | +| MockLoan | `hand-written` (as committed) | baseline | 0 / 21 | 0 / 12 | (not used) | | MockLoan | `hand-written+complete` | baseline | 0 / 21 | 0 / 12 | (threshold baseline) | | MockLoan | `Z-net` | zero-config | no runnable verifier (TODO draft) | | not ready | -| MockLoan | `Z-net-probed` | zero-config | no runnable verifier (refused: the probed read can't see the whole save) | | not ready | +| MockLoan | `Z-net-probed` | zero-config | no runnable verifier (refused: no read covers the whole save; `GET /api` and `GET /api/` return 404, no sibling read) | | not ready | | MockLoan | `Z-net-read` | zero-config (assisted read) | 0 / 21 | 0 / 12 | **ready** | | MockLoan | `Z-db` | zero-config | 0 / 21 | 0 / 12 | **ready** | | MockLoan | `Z-db-gate` | zero-config | 0 / 21 | 0 / 12 | **ready** | @@ -102,8 +119,8 @@ What the generator said about each file it wrote (recorded in - **Labelled draft:** - `Z-net-read` and `P-scaffold-verifier`. Each missed the write-outside-the-read fault, because neither reads the whole store by - itself. `Z-net-probed` now refuses in that case instead of writing a - draft check. + itself. `Z-net-probed` refuses in that case instead of writing a draft + check. - `Z-db-gate`. Two of its fault-gate classes could not be tested on the copy. @@ -113,13 +130,15 @@ What the generator said about each file it wrote (recorded in ## Per fault class +Printed by `report.py` from `results.json`. + ### MockMed -| fault | truth | `hand-written` | `hand-written+complete` | `Z-net-read` | `D-net-read-complete` | `P-record-verify` | `P-scaffold-verifier` | +| fault | truth | `hand-written` | `hand-written+complete` | `D-net-read-complete` | `P-record-verify` | `P-scaffold-verifier` | `Z-net-read` | |---|---|---|---|---|---|---|---| -| ok | correct | 6 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | -| idempotent | correct | 3 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | -| timeout | correct | 3 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | +| ok | correct | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | +| idempotent | correct | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | +| timeout | correct | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | | partial | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | optimistic | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | session | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | @@ -128,15 +147,16 @@ What the generator said about each file it wrote (recorded in | stale | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | no_persist | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | wrong_record | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | -| indeterminate | | 36 | 0 | 0 | 0 | 0 | 0 | +| **total** | | **0 FA / 24, 0 FR / 12** | **0 FA / 24, 0 FR / 12** | **0 FA / 24, 0 FR / 12** | **0 FA / 24, 0 FR / 12** | **0 FA / 24, 0 FR / 12** | **0 FA / 24, 0 FR / 12** | +| indeterminate | | 0 | 0 | 0 | 0 | 0 | 0 | ### MockLoan (held out) -| fault | truth | `hand-written` | `hand-written+complete` | `Z-net-read` | `Z-db` | `Z-db-gate` | `D-net-read-complete` | `P-record-verify` | `P-scaffold-verifier` | +| fault | truth | `hand-written` | `hand-written+complete` | `D-net-read-complete` | `P-record-verify` | `P-scaffold-verifier` | `Z-db` | `Z-db-gate` | `Z-net-read` | |---|---|---|---|---|---|---|---|---|---| -| ok | correct | 6 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | -| idempotent | correct | 3 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | -| timeout | correct | 3 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | +| ok | correct | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | 0 FR / 6 | +| idempotent | correct | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | +| timeout | correct | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | 0 FR / 3 | | partial | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | optimistic | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | session | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | @@ -144,7 +164,8 @@ What the generator said about each file it wrote (recorded in | double | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | stale | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | | collateral | wrong | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | 0 FA / 3 | -| indeterminate | | 33 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | +| **total** | | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | **0 FA / 21, 0 FR / 12** | +| indeterminate | | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | ## What the fixes did, gap by gap @@ -169,19 +190,32 @@ What the generator said about each file it wrote (recorded in Every probe was a `GET` to the local fault server, sent right after the demonstration's save. The probe list is in `results.json` under `apps..arms.Z-net-probed.read_discovery`. +4. **A read that covers the whole save (implemented; nothing to find on + these apps).** Before it refuses, `--probe-reads` now probes the sibling + collections the app read and the prefix root. It uses the root only when + all of these hold: + - the root holds the written record and every record of the collection + and each sibling; + - it is one unpaged response, and every captured write went under the + prefix; + - every top-level value is a list of objects or a scalar. + + The check then reads every list, empty ones included, plus any list that + appears later. On MockLoan the root returns 404 and the app reads no + sibling, so nothing was promoted. The promotion path is covered by tests + on separate local apps (#2147), not by this benchmark. ## Gaps that remain - **A probed read sees one collection, not the store.** On a multi-surface - store, the generator now refuses instead of emitting a check that could - hide a write elsewhere. That is safe, but it means `Z-net-probed` produces - nothing runnable there unless one of these is given: + store, the generator refuses instead of emitting a check that could hide a + write elsewhere. It now looks for a covering read first, but finds one only + when the app reads a sibling collection or serves a JSON prefix root. + MockLoan does neither, so `Z-net-probed` produces nothing runnable there + unless one of these is given: - `--complete-read`; - `--db`; - a fuller read. - - Probing for a sibling read that covers more of the store is not - implemented (follow-up). - **Apps with no JSON read route.** MockMed shows the saved record only as server-rendered HTML (`/chart/`). Discovery reads JSON only, so the read stays unavailable there. Reading an HTML record page is not @@ -207,10 +241,19 @@ What the generator said about each file it wrote (recorded in - the Chrome extension's capture and `benchmark/contract_autogen` were not measured. +## Third run (the refusal, before the covering search) + +That run is in [results-run3.json](results-run3.json), made at cfa11472b. +Every generated arm matched this run. `Z-net-probed` was refused on MockLoan +for the same reason, without the covering search. The one difference is the +`hand-written` (as committed) baseline, which returned INDETERMINATE on all +69 trials (12 of 12 false rejects on each app), because the committed files +then lacked `records_complete: true`. + ## Second run (fixed generator, before the refusal) That run is in [results-run2.json](results-run2.json), made at 0fa5f2b35. -Every arm matched this run except `Z-net-probed` on MockLoan. There, the +Every generated arm matched this run except `Z-net-probed` on MockLoan. There, the generator emitted a draft check on the probed `/api/disbursements` read. With the confirmation flags deleted, it gave 3 false accepts in 21 wrong runs (all `collateral`) and 0 false rejects in 12 correct runs, which is not ready. @@ -233,9 +276,11 @@ hand-written verifiers exactly: 0 false accepts and 0 false rejects. ## A finding outside the generated verifiers -The committed `environments/openenv/{mockmed,mockloan}/verify.yaml` files -still lack `records_complete: true`. They return INDETERMINATE on every run -(69 of 69 here). #1784 adds the line. +In the first three runs, the committed +`environments/openenv/{mockmed,mockloan}/verify.yaml` files lacked +`records_complete: true` and returned INDETERMINATE on every run. #1784 added +the line, and in this run they return the same verdicts as +`hand-written+complete`. ## Reproduce diff --git a/packages/flow/benchmark/zero_config_verify/results-run3.json b/packages/flow/benchmark/zero_config_verify/results-run3.json new file mode 100644 index 000000000..6c3ca15f1 --- /dev/null +++ b/packages/flow/benchmark/zero_config_verify/results-run3.json @@ -0,0 +1,4377 @@ +{ + "apps": { + "mockloan": { + "arms": { + "D-net-read-complete": { + "class": "declared", + "generated_file": "generated/mockloan.D-net-read-complete.yaml", + "generator_misses": [], + "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.826) -> saved record\n read api_db.content: GET /api/db (t=3.826), compared by content -> no record changed or removed\nFault test: 12 of 12 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", + "generator_status": "working", + "inputs": [ + "Z-net-read inputs", + "--complete-read (a person's statement)" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "P-record-verify": { + "class": "read-path-given", + "generated_file": "generated/mockloan.P-record-verify.yaml", + "generator_misses": [], + "generator_output": "", + "generator_status": "proposed", + "inputs": [ + "verifier: read path from the hand-written verify.yaml", + "before/after reads around the demonstration", + "meta.json params" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "P-scaffold-verifier": { + "class": "read-path-given", + "generated_file": "generated/mockloan.P-scaffold-verifier.yaml", + "generator_misses": [ + "collateral_unaudited: a stray row written to a table this check does not read (the check reads only rest_read; a write anywhere else passes it (give a database read path with --db, or more reads))" + ], + "generator_output": "Record check: DRAFT (not working)\n read rest_read: the rest verifier section -> saved record\n read rest_read.content: the rest verifier section, compared by content -> no record changed or removed\nFault test: 12 of 13 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only rest_read; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-P-scaffold-verifier/record_check.proposed.yaml and /out-P-scaffold-verifier/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_status": "draft", + "inputs": [ + "verifier: read path from the hand-written verify.yaml", + "events.jsonl with sor_before/sor_after", + "sor_snapshots.json", + "meta.json params", + "network.jsonl" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "Z-db": { + "class": "zero-config", + "generated_file": "generated/mockloan.Z-db.yaml", + "generator_misses": [], + "generator_output": "Record check: WORKING (not approved)\n read metadata: table metadata -> no row added\n read records: table records -> saved record\n read records.content: table records, compared by content -> no record changed or removed\n read records_all: table records, every row -> saved record\n read records_all.content: table records, every row, compared by content -> no record changed or removed\nFault test: 27 of 27 bad writes caught; clean save passed\nWrote /out-Z-db/record_check.proposed.yaml and /out-Z-db/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", + "generator_status": "working", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl", + "--db " + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "Z-db-gate": { + "class": "zero-config", + "generated_file": "generated/mockloan.Z-db-gate.yaml", + "generator_misses": [ + "fault gate unrelated_edit: another record in the slice edited (the copy holds no other record to edit) (untestable)", + "fault gate side_write: a row in another table added for the saved record (no subject_field: which column of the other tables names the saved record is unknown) (untestable)" + ], + "generator_output": "Record check: DRAFT (not working)\n read metadata: table metadata -> no row added\n read records: table records -> saved record\n read records.content: table records, compared by content -> no record changed or removed\n read records_all: table records, every row -> saved record\n read records_all.content: table records, every row, compared by content -> no record changed or removed\nFault test: 27 of 27 bad writes caught; clean save passed\n MISS: fault gate unrelated_edit: another record in the slice edited (the copy holds no other record to edit) (untestable)\n MISS: fault gate side_write: a row in another table added for the saved record (no subject_field: which column of the other tables names the saved record is unknown) (untestable)\nWrote /out-Z-db-gate/record_check.proposed.yaml and /out-Z-db-gate/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_status": "draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl", + "--db ", + "--fault-gate" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "Z-net": { + "class": "zero-config", + "generated_file": "generated/mockloan.Z-net.yaml", + "generator_misses": [], + "generator_output": "No read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", + "generator_status": "todo-draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app's own traffic)" + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": "VerifyRunError: invalid effect_contract.yaml: 7 validation errors for VerifyConfig\nverifier\n Field required [type=missing, input_value={'workflow': 'rec-plain',...cords_key': 'records'}}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/missing\nworkflow\n Extra inputs are not permitted [type=extra_forbidden, input_value='rec-plain', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource\n Extra inputs are not permitted [type=extra_forbidden, input_value='', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource_kind\n Extra inputs are not permitted [type=extra_forbidden, input_value='recording', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndraft_status\n Extra inputs are not permitted [type=extra_forbidden, input_value='requires-human-edit', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsteps\n Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step_id': 'step_007', ...uthorize Disbursement'}], input_type=list]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndeployment\n Extra inputs are not permitted [type=extra_forbidden, input_value={'effects': {'kind': 'res...ecords_key': 'records'}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden", + "read_discovery": null, + "runnable": false + }, + "Z-net-probed": { + "class": "zero-config", + "generated_file": "generated/mockloan.Z-net-probed.yaml", + "generator_misses": [], + "generator_output": "Read discovery: REFUSED. A probed read confirmed the written values but can't see the whole save, so no check was written: the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read (/out-Z-net-probed/read_discovery.json).\nNo read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net-probed/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net-probed/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", + "generator_status": "todo-draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app's own traffic)", + "--probe-reads (GET probes on the local fixture after the save)" + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": "VerifyRunError: invalid effect_contract.yaml: 7 validation errors for VerifyConfig\nverifier\n Field required [type=missing, input_value={'workflow': 'rec-plain',...cords_key': 'records'}}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/missing\nworkflow\n Extra inputs are not permitted [type=extra_forbidden, input_value='rec-plain', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource\n Extra inputs are not permitted [type=extra_forbidden, input_value='', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource_kind\n Extra inputs are not permitted [type=extra_forbidden, input_value='recording', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndraft_status\n Extra inputs are not permitted [type=extra_forbidden, input_value='requires-human-edit', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsteps\n Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step_id': 'step_007', ...uthorize Disbursement'}], input_type=list]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndeployment\n Extra inputs are not permitted [type=extra_forbidden, input_value={'effects': {'kind': 'res...ecords_key': 'records'}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden", + "read_discovery": { + "method": "GET only, local machine only, after the save", + "needs_operator_confirmation": true, + "probes": [ + { + "kind": "collection", + "method": "GET", + "outcome": "no read here", + "path": "/api/disbursement", + "status": 404 + }, + { + "confirmed_fields": [ + "amount", + "loan_id", + "memo", + "product" + ], + "kind": "collection", + "method": "GET", + "outcome": "confirmed: holds a record with every written value", + "path": "/api/disbursements", + "status": 200 + } + ], + "read": "/api/disbursements", + "reason": "the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read", + "schema": "openadapt.read-discovery/v1", + "status": "refused", + "surface": "api_disbursements" + }, + "runnable": false + }, + "Z-net-read": { + "class": "zero-config", + "generated_file": "generated/mockloan.Z-net-read.yaml", + "generator_misses": [ + "collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))" + ], + "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.826) -> saved record\n read api_db.content: GET /api/db (t=3.826), compared by content -> no record changed or removed\nFault test: 12 of 13 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_status": "draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app traffic plus GET /api/db before and after the save)" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + } + }, + "hand_written": { + "file": "environments/openenv/mockloan/verify.yaml", + "tally": { + "per_fault": { + "collateral": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "double": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "false_reject": 3, + "indeterminate": 3, + "runs": 3 + }, + "ok": { + "false_reject": 6, + "indeterminate": 6, + "runs": 6 + }, + "optimistic": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "partial": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "session": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "stale": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "false_reject": 3, + "indeterminate": 3, + "runs": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 12, + "indeterminate": 33, + "runs": 33, + "true_accept": 0, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "hand_written_complete": { + "file": "the same file plus records_complete: true", + "tally": { + "per_fault": { + "collateral": { + "runs": 3, + "true_reject": 3 + }, + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 33, + "true_accept": 12, + "true_reject": 21, + "wrong_runs": 21 + } + } + }, + "held_out": true, + "trials": [ + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Vehicle purchase, dealer invoice 4471", + "product": "Auto" + }, + "seconds": 8.96, + "seed_concurrent": false, + "task": "mockloan-01", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Home repairs, roof contractor", + "product": "Personal" + }, + "seconds": 9.0, + "seed_concurrent": false, + "task": "mockloan-02", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Second tranche, title received", + "product": "Auto" + }, + "seconds": 8.97, + "seed_concurrent": false, + "task": "mockloan-03", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Debt consolidation, card payoff", + "product": "Personal" + }, + "seconds": 8.97, + "seed_concurrent": true, + "task": "mockloan-04", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Used car, private sale", + "product": "Auto" + }, + "seconds": 8.89, + "seed_concurrent": true, + "task": "mockloan-05", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Tuition payment, spring term", + "product": "Personal" + }, + "seconds": 9.06, + "seed_concurrent": true, + "task": "mockloan-06", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Lease buyout, final payment", + "product": "Auto" + }, + "seconds": 8.96, + "seed_concurrent": false, + "task": "mockloan-07", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Medical bills, clinic invoice", + "product": "Personal" + }, + "seconds": 9.0, + "seed_concurrent": false, + "task": "mockloan-08", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Motorcycle purchase", + "product": "Auto" + }, + "seconds": 8.96, + "seed_concurrent": true, + "task": "mockloan-09", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Truck down payment", + "product": "Auto" + }, + "seconds": 9.01, + "seed_concurrent": false, + "task": "mockloan-10", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Wedding venue deposit", + "product": "Personal" + }, + "seconds": 8.95, + "seed_concurrent": false, + "task": "mockloan-11", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Moving costs, freight quote", + "product": "Personal" + }, + "seconds": 8.99, + "seed_concurrent": true, + "task": "mockloan-12", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-db-gate": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's memo is not the task's value", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Furniture order 2290", + "product": "Personal" + }, + "seconds": 55.36, + "seed_concurrent": false, + "task": "mockloan-13", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's memo is not the task's value", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Camper van purchase", + "product": "Auto" + }, + "seconds": 54.99, + "seed_concurrent": false, + "task": "mockloan-14", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's memo is not the task's value", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Fleet vehicle, unit 7", + "product": "Auto" + }, + "seconds": 55.02, + "seed_concurrent": true, + "task": "mockloan-15", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Dental work, phase one", + "product": "Personal" + }, + "seconds": 65.3, + "seed_concurrent": false, + "task": "mockloan-16", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Electric car, charger included", + "product": "Auto" + }, + "seconds": 65.28, + "seed_concurrent": false, + "task": "mockloan-17", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Boat trailer purchase", + "product": "Auto" + }, + "seconds": 65.18, + "seed_concurrent": true, + "task": "mockloan-18", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Bridge loan, closing costs", + "product": "Personal" + }, + "seconds": 55.01, + "seed_concurrent": false, + "task": "mockloan-19", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Hybrid sedan purchase", + "product": "Auto" + }, + "seconds": 54.8, + "seed_concurrent": false, + "task": "mockloan-20", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Solar panels, installer deposit", + "product": "Personal" + }, + "seconds": 54.6, + "seed_concurrent": true, + "task": "mockloan-21", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Delivery van, invoice 882", + "product": "Auto" + }, + "seconds": 54.94, + "seed_concurrent": false, + "task": "mockloan-22", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Home office equipment", + "product": "Personal" + }, + "seconds": 55.25, + "seed_concurrent": false, + "task": "mockloan-23", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Business licence fees", + "product": "Personal" + }, + "seconds": 55.12, + "seed_concurrent": true, + "task": "mockloan-24", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Pickup truck, trade-in applied", + "product": "Auto" + }, + "seconds": 55.35, + "seed_concurrent": false, + "task": "mockloan-25", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Vet bills, emergency visit", + "product": "Personal" + }, + "seconds": 55.01, + "seed_concurrent": false, + "task": "mockloan-26", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Compact car, cash balance", + "product": "Auto" + }, + "seconds": 54.97, + "seed_concurrent": true, + "task": "mockloan-27", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Renovation draw two", + "product": "Personal" + }, + "seconds": 49.97, + "seed_concurrent": true, + "task": "mockloan-28", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Replacement vehicle after accident", + "product": "Auto" + }, + "seconds": 50.25, + "seed_concurrent": true, + "task": "mockloan-29", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Inventory purchase, spring stock", + "product": "Personal" + }, + "seconds": 50.02, + "seed_concurrent": true, + "task": "mockloan-30", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "collateral", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "32000", + "loan_id": "L1002", + "memo": "Course fees, evening classes", + "product": "Personal" + }, + "seconds": 49.68, + "seed_concurrent": false, + "task": "mockloan-31", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "collateral", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "9500", + "loan_id": "L1003", + "memo": "Classic car restoration", + "product": "Auto" + }, + "seconds": 49.83, + "seed_concurrent": false, + "task": "mockloan-32", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "collateral", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "amount": "18500", + "loan_id": "L1001", + "memo": "Kitchen appliances", + "product": "Personal" + }, + "seconds": 50.01, + "seed_concurrent": true, + "task": "mockloan-33", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db": { + "error": null, + "verdict": "REFUTED" + }, + "Z-db-gate": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + } + ] + }, + "mockmed": { + "arms": { + "D-net-read-complete": { + "class": "declared", + "generated_file": "generated/mockmed.D-net-read-complete.yaml", + "generator_misses": [], + "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.769) -> saved record\n read api_db.content: GET /api/db (t=3.769), compared by content -> no record changed or removed\nFault test: 10 of 10 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", + "generator_status": "working", + "inputs": [ + "Z-net-read inputs", + "--complete-read (a person's statement)" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "no_persist": { + "runs": 3, + "true_reject": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + }, + "wrong_record": { + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 36, + "true_accept": 12, + "true_reject": 24, + "wrong_runs": 24 + } + } + }, + "P-record-verify": { + "class": "read-path-given", + "generated_file": "generated/mockmed.P-record-verify.yaml", + "generator_misses": [], + "generator_output": "", + "generator_status": "proposed", + "inputs": [ + "verifier: read path from the hand-written verify.yaml", + "before/after reads around the demonstration", + "meta.json params" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "no_persist": { + "runs": 3, + "true_reject": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + }, + "wrong_record": { + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 36, + "true_accept": 12, + "true_reject": 24, + "wrong_runs": 24 + } + } + }, + "P-scaffold-verifier": { + "class": "read-path-given", + "generated_file": "generated/mockmed.P-scaffold-verifier.yaml", + "generator_misses": [ + "collateral_unaudited: a stray row written to a table this check does not read (the check reads only rest_read; a write anywhere else passes it (give a database read path with --db, or more reads))" + ], + "generator_output": "Record check: DRAFT (not working)\n read rest_read: the rest verifier section -> saved record\n read rest_read.content: the rest verifier section, compared by content -> no record changed or removed\nFault test: 10 of 11 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only rest_read; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-P-scaffold-verifier/record_check.proposed.yaml and /out-P-scaffold-verifier/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_status": "draft", + "inputs": [ + "verifier: read path from the hand-written verify.yaml", + "events.jsonl with sor_before/sor_after", + "sor_snapshots.json", + "meta.json params", + "network.jsonl" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "no_persist": { + "runs": 3, + "true_reject": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + }, + "wrong_record": { + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 36, + "true_accept": 12, + "true_reject": 24, + "wrong_runs": 24 + } + } + }, + "Z-db": { + "class": "zero-config", + "generator_misses": [], + "generator_output": "the fault server keeps its store in memory: no database file", + "generator_status": "not-applicable", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl", + "--db " + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": null, + "read_discovery": null, + "runnable": false + }, + "Z-db-gate": { + "class": "zero-config", + "generator_misses": [], + "generator_output": "the fault server keeps its store in memory: no database file", + "generator_status": "not-applicable", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl", + "--db ", + "--fault-gate" + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": null, + "read_discovery": null, + "runnable": false + }, + "Z-net": { + "class": "zero-config", + "generated_file": "generated/mockmed.Z-net.yaml", + "generator_misses": [], + "generator_output": "No read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", + "generator_status": "todo-draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app's own traffic)" + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": "VerifyRunError: invalid effect_contract.yaml: 7 validation errors for VerifyConfig\nverifier\n Field required [type=missing, input_value={'workflow': 'rec-plain',...cords_key': 'records'}}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/missing\nworkflow\n Extra inputs are not permitted [type=extra_forbidden, input_value='rec-plain', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource\n Extra inputs are not permitted [type=extra_forbidden, input_value='', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource_kind\n Extra inputs are not permitted [type=extra_forbidden, input_value='recording', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndraft_status\n Extra inputs are not permitted [type=extra_forbidden, input_value='requires-human-edit', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsteps\n Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step_id': 'step_007', ...ounter Save Encounter'}], input_type=list]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndeployment\n Extra inputs are not permitted [type=extra_forbidden, input_value={'effects': {'kind': 'res...ecords_key': 'records'}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden", + "read_discovery": null, + "runnable": false + }, + "Z-net-probed": { + "class": "zero-config", + "generated_file": "generated/mockmed.Z-net-probed.yaml", + "generator_misses": [], + "generator_output": "Read discovery: UNAVAILABLE. No probed read confirmed the written values, so no read was proposed; a person must name one (/out-Z-net-probed/read_discovery.json).\nNo read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net-probed/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net-probed/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", + "generator_status": "todo-draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app's own traffic)", + "--probe-reads (GET probes on the local fixture after the save)" + ], + "kill_check": { + "failed": [ + "no runnable verifier" + ], + "ready": false + }, + "parse_error": "VerifyRunError: invalid effect_contract.yaml: 7 validation errors for VerifyConfig\nverifier\n Field required [type=missing, input_value={'workflow': 'rec-plain',...cords_key': 'records'}}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/missing\nworkflow\n Extra inputs are not permitted [type=extra_forbidden, input_value='rec-plain', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource\n Extra inputs are not permitted [type=extra_forbidden, input_value='', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource_kind\n Extra inputs are not permitted [type=extra_forbidden, input_value='recording', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndraft_status\n Extra inputs are not permitted [type=extra_forbidden, input_value='requires-human-edit', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsteps\n Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step_id': 'step_007', ...ounter Save Encounter'}], input_type=list]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndeployment\n Extra inputs are not permitted [type=extra_forbidden, input_value={'effects': {'kind': 'res...ecords_key': 'records'}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden", + "read_discovery": { + "method": "GET only, local machine only, after the save", + "needs_operator_confirmation": true, + "probes": [ + { + "kind": "collection", + "method": "GET", + "outcome": "no read here", + "path": "/api/encounter", + "status": 404 + }, + { + "kind": "collection", + "method": "GET", + "outcome": "no read here", + "path": "/api/encounters", + "status": 404 + }, + { + "kind": "item", + "method": "GET", + "outcome": "no read here", + "path": "/api/encounter/1", + "status": 404 + }, + { + "kind": "item", + "method": "GET", + "outcome": "no read here", + "path": "/api/encounters/1", + "status": 404 + } + ], + "reason": "no probed collection read returned a record with every written value; a person must name the read", + "schema": "openadapt.read-discovery/v1", + "status": "unavailable" + }, + "runnable": false + }, + "Z-net-read": { + "class": "zero-config", + "generated_file": "generated/mockmed.Z-net-read.yaml", + "generator_misses": [ + "collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))" + ], + "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.769) -> saved record\n read api_db.content: GET /api/db (t=3.769), compared by content -> no record changed or removed\nFault test: 10 of 11 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_status": "draft", + "inputs": [ + "events.jsonl", + "meta.json params", + "network.jsonl (app traffic plus GET /api/db before and after the save)" + ], + "kill_check": { + "failed": [], + "ready": true + }, + "parse_error": null, + "read_discovery": null, + "runnable": true, + "tally": { + "per_fault": { + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "no_persist": { + "runs": 3, + "true_reject": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + }, + "wrong_record": { + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 36, + "true_accept": 12, + "true_reject": 24, + "wrong_runs": 24 + } + } + } + }, + "hand_written": { + "file": "environments/openenv/mockmed/verify.yaml", + "tally": { + "per_fault": { + "double": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "false_reject": 3, + "indeterminate": 3, + "runs": 3 + }, + "no_persist": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "ok": { + "false_reject": 6, + "indeterminate": 6, + "runs": 6 + }, + "optimistic": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "partial": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "session": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "stale": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "false_reject": 3, + "indeterminate": 3, + "runs": 3 + }, + "wrong_record": { + "indeterminate": 3, + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 12, + "indeterminate": 36, + "runs": 36, + "true_accept": 0, + "true_reject": 24, + "wrong_runs": 24 + } + } + }, + "hand_written_complete": { + "file": "the same file plus records_complete: true", + "tally": { + "per_fault": { + "double": { + "runs": 3, + "true_reject": 3 + }, + "duplicate": { + "runs": 3, + "true_reject": 3 + }, + "idempotent": { + "runs": 3, + "true_accept": 3 + }, + "no_persist": { + "runs": 3, + "true_reject": 3 + }, + "ok": { + "runs": 6, + "true_accept": 6 + }, + "optimistic": { + "runs": 3, + "true_reject": 3 + }, + "partial": { + "runs": 3, + "true_reject": 3 + }, + "session": { + "runs": 3, + "true_reject": 3 + }, + "stale": { + "runs": 3, + "true_reject": 3 + }, + "timeout": { + "runs": 3, + "true_accept": 3 + }, + "wrong_record": { + "runs": 3, + "true_reject": 3 + } + }, + "total": { + "correct_runs": 12, + "false_accept": 0, + "false_reject": 0, + "indeterminate": 0, + "runs": 36, + "true_accept": 12, + "true_reject": 24, + "wrong_runs": 24 + } + } + }, + "held_out": false, + "trials": [ + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Shoulder strain, physio twice weekly", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 9.01, + "seed_concurrent": false, + "task": "mockmed-01", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Mild fever overnight, recheck tomorrow", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 8.95, + "seed_concurrent": false, + "task": "mockmed-02", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Asthma review, inhaler technique checked", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 8.96, + "seed_concurrent": false, + "task": "mockmed-03", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Ankle sprain, compression and elevation", + "patient_id": "p2", + "type": "Triage" + }, + "seconds": 8.93, + "seed_concurrent": true, + "task": "mockmed-04", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Eczema flare, steroid cream for 5 days", + "patient_id": "p3", + "type": "Consult" + }, + "seconds": 8.94, + "seed_concurrent": true, + "task": "mockmed-05", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "ok", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Headache resolved, no further action", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 8.94, + "seed_concurrent": true, + "task": "mockmed-06", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Blood pressure log reviewed, no change", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 8.98, + "seed_concurrent": false, + "task": "mockmed-07", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Insect bite, antihistamine advised", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 8.96, + "seed_concurrent": false, + "task": "mockmed-08", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "idempotent", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Sleep diary reviewed, follow up in 4 weeks", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 8.93, + "seed_concurrent": true, + "task": "mockmed-09", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Thyroid results normal, annual recheck", + "patient_id": "p3", + "type": "Consult" + }, + "seconds": 9.02, + "seed_concurrent": false, + "task": "mockmed-10", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Wrist pain after fall, x-ray requested", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 8.94, + "seed_concurrent": false, + "task": "mockmed-11", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": true, + "fault": "timeout", + "ground_truth_correct": true, + "ground_truth_reason": "exactly one new record with the task's values", + "label_agrees": true, + "params": { + "note": "Cough for 3 days, no red flags", + "patient_id": "p2", + "type": "Triage" + }, + "seconds": 8.92, + "seed_concurrent": true, + "task": "mockmed-12", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-record-verify": { + "error": null, + "verdict": "CONFIRMED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "CONFIRMED" + }, + "Z-net-read": { + "error": null, + "verdict": "CONFIRMED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "CONFIRMED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's note is not the task's value", + "label_agrees": true, + "params": { + "note": "Back pain, heat pack and gentle stretches", + "patient_id": "p2", + "type": "Triage" + }, + "seconds": 34.86, + "seed_concurrent": false, + "task": "mockmed-13", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's note is not the task's value", + "label_agrees": true, + "params": { + "note": "Migraine plan updated, triptan as needed", + "patient_id": "p3", + "type": "Consult" + }, + "seconds": 34.74, + "seed_concurrent": false, + "task": "mockmed-14", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "partial", + "ground_truth_correct": false, + "ground_truth_reason": "new record's note is not the task's value", + "label_agrees": true, + "params": { + "note": "Hay fever, nasal spray daily", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 34.79, + "seed_concurrent": true, + "task": "mockmed-15", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Nosebleed stopped with pressure", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 34.86, + "seed_concurrent": false, + "task": "mockmed-16", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Vitamin D low, supplement started", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 34.84, + "seed_concurrent": false, + "task": "mockmed-17", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "optimistic", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Knee brace fitted, review in 6 weeks", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 34.81, + "seed_concurrent": true, + "task": "mockmed-18", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Earache, drops for one week", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 34.3, + "seed_concurrent": false, + "task": "mockmed-19", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Diet review, dietitian referral", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 34.79, + "seed_concurrent": false, + "task": "mockmed-20", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "session", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Minor burn dressed, keep dry", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 34.76, + "seed_concurrent": true, + "task": "mockmed-21", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Anaemia follow-up, iron continued", + "patient_id": "p3", + "type": "Consult" + }, + "seconds": 34.81, + "seed_concurrent": false, + "task": "mockmed-22", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Sinus pressure, saline rinse", + "patient_id": "p2", + "type": "Triage" + }, + "seconds": 34.76, + "seed_concurrent": false, + "task": "mockmed-23", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "duplicate", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Bruised rib, simple analgesia", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 34.72, + "seed_concurrent": true, + "task": "mockmed-24", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Cholesterol recheck in 3 months", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 34.8, + "seed_concurrent": false, + "task": "mockmed-25", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Splinter removed, wound clean", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 34.77, + "seed_concurrent": false, + "task": "mockmed-26", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "double", + "ground_truth_correct": false, + "ground_truth_reason": "2 new record(s)", + "label_agrees": true, + "params": { + "note": "Smoking cessation plan agreed", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 34.76, + "seed_concurrent": true, + "task": "mockmed-27", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "note": "Repeat prescription for antihistamine", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 50.31, + "seed_concurrent": true, + "task": "mockmed-28", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "note": "Physio progress good, continue exercises", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 50.25, + "seed_concurrent": true, + "task": "mockmed-29", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "stale", + "ground_truth_correct": false, + "ground_truth_reason": "1 existing record(s) lost or changed", + "label_agrees": true, + "params": { + "note": "Dizziness on standing, fluids advised", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 50.22, + "seed_concurrent": true, + "task": "mockmed-30", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "no_persist", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Sunburn, aloe and shade", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 34.83, + "seed_concurrent": false, + "task": "mockmed-31", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "no_persist", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Hearing test booked", + "patient_id": "p2", + "type": "Consult" + }, + "seconds": 34.92, + "seed_concurrent": false, + "task": "mockmed-32", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "no_persist", + "ground_truth_correct": false, + "ground_truth_reason": "0 new record(s)", + "label_agrees": true, + "params": { + "note": "Mouth ulcer, salt rinse", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 34.74, + "seed_concurrent": true, + "task": "mockmed-33", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "wrong_record", + "ground_truth_correct": false, + "ground_truth_reason": "new record's patient_id is not the task's value", + "label_agrees": true, + "params": { + "note": "Gout flare, colchicine short course", + "patient_id": "p1", + "type": "Consult" + }, + "seconds": 34.75, + "seed_concurrent": false, + "task": "mockmed-34", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "wrong_record", + "ground_truth_correct": false, + "ground_truth_reason": "new record's patient_id is not the task's value", + "label_agrees": true, + "params": { + "note": "Stye on left eyelid, warm compress", + "patient_id": "p3", + "type": "Triage" + }, + "seconds": 34.71, + "seed_concurrent": false, + "task": "mockmed-35", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + }, + { + "design_label_correct": false, + "fault": "wrong_record", + "ground_truth_correct": false, + "ground_truth_reason": "new record's patient_id is not the task's value", + "label_agrees": true, + "params": { + "note": "Hiccups resolved without treatment", + "patient_id": "p1", + "type": "Triage" + }, + "seconds": 34.7, + "seed_concurrent": true, + "task": "mockmed-36", + "verdicts": { + "D-net-read-complete": { + "error": null, + "verdict": "REFUTED" + }, + "P-record-verify": { + "error": null, + "verdict": "REFUTED" + }, + "P-scaffold-verifier": { + "error": null, + "verdict": "REFUTED" + }, + "Z-net-read": { + "error": null, + "verdict": "REFUTED" + }, + "hand-written": { + "error": null, + "verdict": "INDETERMINATE" + }, + "hand-written+complete": { + "error": null, + "verdict": "REFUTED" + } + } + } + ] + } + }, + "git_head": "cfa11472b7cb656764577b599ddf66874db446bc", + "platform": "macOS-15.7.3-arm64-arm-64bit", + "protocol": "PROTOCOL.md", + "python": "3.12.13", + "schema": "openadapt.zero-config-verify-results/v1", + "started": "2026-11-09T17:28:57Z", + "verdict": { + "zero_config_ready_arms": { + "mockloan": [ + "Z-db", + "Z-db-gate", + "Z-net-read" + ], + "mockmed": [ + "Z-net-read" + ] + }, + "zero_config_ready_on_both_apps": true + } +} diff --git a/packages/flow/benchmark/zero_config_verify/results.json b/packages/flow/benchmark/zero_config_verify/results.json index 6c3ca15f1..458237471 100644 --- a/packages/flow/benchmark/zero_config_verify/results.json +++ b/packages/flow/benchmark/zero_config_verify/results.json @@ -6,7 +6,7 @@ "class": "declared", "generated_file": "generated/mockloan.D-net-read-complete.yaml", "generator_misses": [], - "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.826) -> saved record\n read api_db.content: GET /api/db (t=3.826), compared by content -> no record changed or removed\nFault test: 12 of 12 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", + "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.767) -> saved record\n read api_db.content: GET /api/db (t=3.767), compared by content -> no record changed or removed\nFault test: 12 of 12 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", "generator_status": "working", "inputs": [ "Z-net-read inputs", @@ -401,7 +401,7 @@ "class": "zero-config", "generated_file": "generated/mockloan.Z-net-probed.yaml", "generator_misses": [], - "generator_output": "Read discovery: REFUSED. A probed read confirmed the written values but can't see the whole save, so no check was written: the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read (/out-Z-net-probed/read_discovery.json).\nNo read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net-probed/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net-probed/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", + "generator_output": "Read discovery: REFUSED. A probed read confirmed the written values but can't see the whole save, so no check was written: the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No read covering the whole save was found: GET /api: no read here (status 404); GET /api/: no read here (status 404); the app read no other collection under /api. No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read (/out-Z-net-probed/read_discovery.json).\nNo read in the recording returns the saved record (no network capture, --db or --verifier), so this is a TODO draft.\nDraft written to /out-Z-net-probed/effect_contract.yaml (1 write-shaped step(s))\n\nThis is a DRAFT oracle, not an approved contract: it verifies nothing until a\nperson resolves every TODO, wires the deployment section, and qualifies the\nbundle.\n\nNext commands:\n 1. edit /out-Z-net-probed/effect_contract.yaml\n 2. openadapt-flow lint \n 3. openadapt-flow certify --policy \n 4. openadapt-flow qualify init --target ...", "generator_status": "todo-draft", "inputs": [ "events.jsonl", @@ -417,6 +417,10 @@ }, "parse_error": "VerifyRunError: invalid effect_contract.yaml: 7 validation errors for VerifyConfig\nverifier\n Field required [type=missing, input_value={'workflow': 'rec-plain',...cords_key': 'records'}}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/missing\nworkflow\n Extra inputs are not permitted [type=extra_forbidden, input_value='rec-plain', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource\n Extra inputs are not permitted [type=extra_forbidden, input_value='', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsource_kind\n Extra inputs are not permitted [type=extra_forbidden, input_value='recording', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndraft_status\n Extra inputs are not permitted [type=extra_forbidden, input_value='requires-human-edit', input_type=str]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\nsteps\n Extra inputs are not permitted [type=extra_forbidden, input_value=[{'step_id': 'step_007', ...uthorize Disbursement'}], input_type=list]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden\ndeployment\n Extra inputs are not permitted [type=extra_forbidden, input_value={'effects': {'kind': 'res...ecords_key': 'records'}}, input_type=dict]\n For further information visit https://errors.pydantic.dev/2.13/v/extra_forbidden", "read_discovery": { + "covering_read": { + "reason": "GET /api: no read here (status 404); GET /api/: no read here (status 404); the app read no other collection under /api", + "status": "not found" + }, "method": "GET only, local machine only, after the save", "needs_operator_confirmation": true, "probes": [ @@ -439,10 +443,26 @@ "outcome": "confirmed: holds a record with every written value", "path": "/api/disbursements", "status": 200 + }, + { + "kind": "root", + "method": "GET", + "outcome": "no read here", + "path": "/api", + "purpose": "a read that covers the whole save", + "status": 404 + }, + { + "kind": "root", + "method": "GET", + "outcome": "no read here", + "path": "/api/", + "purpose": "a read that covers the whole save", + "status": 404 } ], "read": "/api/disbursements", - "reason": "the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read", + "reason": "the probed read /api/disbursements covers only the written collection, and a write outside it would pass the check (collateral_unaudited: a stray row written to a table this check does not read). No read covering the whole save was found: GET /api: no read here (status 404); GET /api/: no read here (status 404); the app read no other collection under /api. No runnable check was written; confirm the read covers everything the save can change (--complete-read), give --db, or name a fuller read", "schema": "openadapt.read-discovery/v1", "status": "refused", "surface": "api_disbursements" @@ -455,7 +475,7 @@ "generator_misses": [ "collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))" ], - "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.826) -> saved record\n read api_db.content: GET /api/db (t=3.826), compared by content -> no record changed or removed\nFault test: 12 of 13 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.767) -> saved record\n read api_db.content: GET /api/db (t=3.767), compared by content -> no record changed or removed\nFault test: 12 of 13 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", "generator_status": "draft", "inputs": [ "events.jsonl", @@ -530,63 +550,53 @@ "tally": { "per_fault": { "collateral": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "double": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "duplicate": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "idempotent": { - "false_reject": 3, - "indeterminate": 3, - "runs": 3 + "runs": 3, + "true_accept": 3 }, "ok": { - "false_reject": 6, - "indeterminate": 6, - "runs": 6 + "runs": 6, + "true_accept": 6 }, "optimistic": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "partial": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "session": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "stale": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "timeout": { - "false_reject": 3, - "indeterminate": 3, - "runs": 3 + "runs": 3, + "true_accept": 3 } }, "total": { "correct_runs": 12, "false_accept": 0, - "false_reject": 12, - "indeterminate": 33, + "false_reject": 0, + "indeterminate": 0, "runs": 33, - "true_accept": 0, + "true_accept": 12, "true_reject": 21, "wrong_runs": 21 } @@ -663,7 +673,7 @@ "memo": "Vehicle purchase, dealer invoice 4471", "product": "Auto" }, - "seconds": 8.96, + "seconds": 3.79, "seed_concurrent": false, "task": "mockloan-01", "verdicts": { @@ -693,7 +703,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -713,7 +723,7 @@ "memo": "Home repairs, roof contractor", "product": "Personal" }, - "seconds": 9.0, + "seconds": 3.78, "seed_concurrent": false, "task": "mockloan-02", "verdicts": { @@ -743,7 +753,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -763,7 +773,7 @@ "memo": "Second tranche, title received", "product": "Auto" }, - "seconds": 8.97, + "seconds": 3.8, "seed_concurrent": false, "task": "mockloan-03", "verdicts": { @@ -793,7 +803,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -813,7 +823,7 @@ "memo": "Debt consolidation, card payoff", "product": "Personal" }, - "seconds": 8.97, + "seconds": 3.8, "seed_concurrent": true, "task": "mockloan-04", "verdicts": { @@ -843,7 +853,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -863,7 +873,7 @@ "memo": "Used car, private sale", "product": "Auto" }, - "seconds": 8.89, + "seconds": 3.77, "seed_concurrent": true, "task": "mockloan-05", "verdicts": { @@ -893,7 +903,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -913,7 +923,7 @@ "memo": "Tuition payment, spring term", "product": "Personal" }, - "seconds": 9.06, + "seconds": 3.78, "seed_concurrent": true, "task": "mockloan-06", "verdicts": { @@ -943,7 +953,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -963,7 +973,7 @@ "memo": "Lease buyout, final payment", "product": "Auto" }, - "seconds": 8.96, + "seconds": 3.77, "seed_concurrent": false, "task": "mockloan-07", "verdicts": { @@ -993,7 +1003,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1013,7 +1023,7 @@ "memo": "Medical bills, clinic invoice", "product": "Personal" }, - "seconds": 9.0, + "seconds": 3.78, "seed_concurrent": false, "task": "mockloan-08", "verdicts": { @@ -1043,7 +1053,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1063,7 +1073,7 @@ "memo": "Motorcycle purchase", "product": "Auto" }, - "seconds": 8.96, + "seconds": 3.84, "seed_concurrent": true, "task": "mockloan-09", "verdicts": { @@ -1093,7 +1103,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1113,7 +1123,7 @@ "memo": "Truck down payment", "product": "Auto" }, - "seconds": 9.01, + "seconds": 3.79, "seed_concurrent": false, "task": "mockloan-10", "verdicts": { @@ -1143,7 +1153,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1163,7 +1173,7 @@ "memo": "Wedding venue deposit", "product": "Personal" }, - "seconds": 8.95, + "seconds": 3.79, "seed_concurrent": false, "task": "mockloan-11", "verdicts": { @@ -1193,7 +1203,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1213,7 +1223,7 @@ "memo": "Moving costs, freight quote", "product": "Personal" }, - "seconds": 8.99, + "seconds": 3.82, "seed_concurrent": true, "task": "mockloan-12", "verdicts": { @@ -1243,7 +1253,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -1263,7 +1273,7 @@ "memo": "Furniture order 2290", "product": "Personal" }, - "seconds": 55.36, + "seconds": 55.3, "seed_concurrent": false, "task": "mockloan-13", "verdicts": { @@ -1293,7 +1303,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1313,7 +1323,7 @@ "memo": "Camper van purchase", "product": "Auto" }, - "seconds": 54.99, + "seconds": 55.13, "seed_concurrent": false, "task": "mockloan-14", "verdicts": { @@ -1343,7 +1353,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1363,7 +1373,7 @@ "memo": "Fleet vehicle, unit 7", "product": "Auto" }, - "seconds": 55.02, + "seconds": 55.15, "seed_concurrent": true, "task": "mockloan-15", "verdicts": { @@ -1393,7 +1403,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1413,7 +1423,7 @@ "memo": "Dental work, phase one", "product": "Personal" }, - "seconds": 65.3, + "seconds": 65.35, "seed_concurrent": false, "task": "mockloan-16", "verdicts": { @@ -1443,7 +1453,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1463,7 +1473,7 @@ "memo": "Electric car, charger included", "product": "Auto" }, - "seconds": 65.28, + "seconds": 65.34, "seed_concurrent": false, "task": "mockloan-17", "verdicts": { @@ -1493,7 +1503,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1513,7 +1523,7 @@ "memo": "Boat trailer purchase", "product": "Auto" }, - "seconds": 65.18, + "seconds": 65.36, "seed_concurrent": true, "task": "mockloan-18", "verdicts": { @@ -1543,7 +1553,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1563,7 +1573,7 @@ "memo": "Bridge loan, closing costs", "product": "Personal" }, - "seconds": 55.01, + "seconds": 54.97, "seed_concurrent": false, "task": "mockloan-19", "verdicts": { @@ -1593,7 +1603,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1613,7 +1623,7 @@ "memo": "Hybrid sedan purchase", "product": "Auto" }, - "seconds": 54.8, + "seconds": 54.79, "seed_concurrent": false, "task": "mockloan-20", "verdicts": { @@ -1643,7 +1653,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1663,7 +1673,7 @@ "memo": "Solar panels, installer deposit", "product": "Personal" }, - "seconds": 54.6, + "seconds": 54.71, "seed_concurrent": true, "task": "mockloan-21", "verdicts": { @@ -1693,7 +1703,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1713,7 +1723,7 @@ "memo": "Delivery van, invoice 882", "product": "Auto" }, - "seconds": 54.94, + "seconds": 54.75, "seed_concurrent": false, "task": "mockloan-22", "verdicts": { @@ -1743,7 +1753,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1763,7 +1773,7 @@ "memo": "Home office equipment", "product": "Personal" }, - "seconds": 55.25, + "seconds": 54.91, "seed_concurrent": false, "task": "mockloan-23", "verdicts": { @@ -1793,7 +1803,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1813,7 +1823,7 @@ "memo": "Business licence fees", "product": "Personal" }, - "seconds": 55.12, + "seconds": 54.68, "seed_concurrent": true, "task": "mockloan-24", "verdicts": { @@ -1843,7 +1853,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1863,7 +1873,7 @@ "memo": "Pickup truck, trade-in applied", "product": "Auto" }, - "seconds": 55.35, + "seconds": 54.63, "seed_concurrent": false, "task": "mockloan-25", "verdicts": { @@ -1893,7 +1903,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1913,7 +1923,7 @@ "memo": "Vet bills, emergency visit", "product": "Personal" }, - "seconds": 55.01, + "seconds": 54.65, "seed_concurrent": false, "task": "mockloan-26", "verdicts": { @@ -1943,7 +1953,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -1963,7 +1973,7 @@ "memo": "Compact car, cash balance", "product": "Auto" }, - "seconds": 54.97, + "seconds": 55.11, "seed_concurrent": true, "task": "mockloan-27", "verdicts": { @@ -1993,7 +2003,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2013,7 +2023,7 @@ "memo": "Renovation draw two", "product": "Personal" }, - "seconds": 49.97, + "seconds": 49.35, "seed_concurrent": true, "task": "mockloan-28", "verdicts": { @@ -2043,7 +2053,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2063,7 +2073,7 @@ "memo": "Replacement vehicle after accident", "product": "Auto" }, - "seconds": 50.25, + "seconds": 49.06, "seed_concurrent": true, "task": "mockloan-29", "verdicts": { @@ -2093,7 +2103,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2113,7 +2123,7 @@ "memo": "Inventory purchase, spring stock", "product": "Personal" }, - "seconds": 50.02, + "seconds": 49.22, "seed_concurrent": true, "task": "mockloan-30", "verdicts": { @@ -2143,7 +2153,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2163,7 +2173,7 @@ "memo": "Course fees, evening classes", "product": "Personal" }, - "seconds": 49.68, + "seconds": 49.88, "seed_concurrent": false, "task": "mockloan-31", "verdicts": { @@ -2193,7 +2203,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2213,7 +2223,7 @@ "memo": "Classic car restoration", "product": "Auto" }, - "seconds": 49.83, + "seconds": 49.66, "seed_concurrent": false, "task": "mockloan-32", "verdicts": { @@ -2243,7 +2253,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2263,7 +2273,7 @@ "memo": "Kitchen appliances", "product": "Personal" }, - "seconds": 50.01, + "seconds": 49.7, "seed_concurrent": true, "task": "mockloan-33", "verdicts": { @@ -2293,7 +2303,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -2309,7 +2319,7 @@ "class": "declared", "generated_file": "generated/mockmed.D-net-read-complete.yaml", "generator_misses": [], - "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.769) -> saved record\n read api_db.content: GET /api/db (t=3.769), compared by content -> no record changed or removed\nFault test: 10 of 10 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", + "generator_output": "Record check: WORKING (not approved)\n read api_db: GET /api/db (t=3.787) -> saved record\n read api_db.content: GET /api/db (t=3.787), compared by content -> no record changed or removed\nFault test: 10 of 10 bad writes caught; clean save passed\nWrote /out-D-net-read-complete/record_check.proposed.yaml and /out-D-net-read-complete/record_check.proof.json\nNext: a person reviews it and deletes each `needs_operator_confirmation: true` line, then `openadapt-flow qualify --verify record_check.proposed.yaml`.", "generator_status": "working", "inputs": [ "Z-net-read inputs", @@ -2667,7 +2677,7 @@ "generator_misses": [ "collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))" ], - "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.769) -> saved record\n read api_db.content: GET /api/db (t=3.769), compared by content -> no record changed or removed\nFault test: 10 of 11 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", + "generator_output": "Record check: DRAFT (not working)\n read api_db: GET /api/db (t=3.787) -> saved record\n read api_db.content: GET /api/db (t=3.787), compared by content -> no record changed or removed\nFault test: 10 of 11 bad writes caught; clean save passed\n MISS: collateral_unaudited: a stray row written to a table this check does not read (the check reads only api_db; a write anywhere else passes it (give a database read path with --db, or more reads))\nWrote /out-Z-net-read/record_check.proposed.yaml and /out-Z-net-read/record_check.proof.json\nNext: fix the misses above, then run scaffold-verifier again.", "generator_status": "draft", "inputs": [ "events.jsonl", @@ -2746,57 +2756,46 @@ "tally": { "per_fault": { "double": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "duplicate": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "idempotent": { - "false_reject": 3, - "indeterminate": 3, - "runs": 3 + "runs": 3, + "true_accept": 3 }, "no_persist": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "ok": { - "false_reject": 6, - "indeterminate": 6, - "runs": 6 + "runs": 6, + "true_accept": 6 }, "optimistic": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "partial": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "session": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "stale": { - "indeterminate": 3, "runs": 3, "true_reject": 3 }, "timeout": { - "false_reject": 3, - "indeterminate": 3, - "runs": 3 + "runs": 3, + "true_accept": 3 }, "wrong_record": { - "indeterminate": 3, "runs": 3, "true_reject": 3 } @@ -2804,10 +2803,10 @@ "total": { "correct_runs": 12, "false_accept": 0, - "false_reject": 12, - "indeterminate": 36, + "false_reject": 0, + "indeterminate": 0, "runs": 36, - "true_accept": 0, + "true_accept": 12, "true_reject": 24, "wrong_runs": 24 } @@ -2887,7 +2886,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 9.01, + "seconds": 3.85, "seed_concurrent": false, "task": "mockmed-01", "verdicts": { @@ -2909,7 +2908,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -2928,7 +2927,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 8.95, + "seconds": 3.81, "seed_concurrent": false, "task": "mockmed-02", "verdicts": { @@ -2950,7 +2949,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -2969,7 +2968,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 8.96, + "seconds": 3.83, "seed_concurrent": false, "task": "mockmed-03", "verdicts": { @@ -2991,7 +2990,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3010,7 +3009,7 @@ "patient_id": "p2", "type": "Triage" }, - "seconds": 8.93, + "seconds": 3.8, "seed_concurrent": true, "task": "mockmed-04", "verdicts": { @@ -3032,7 +3031,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3051,7 +3050,7 @@ "patient_id": "p3", "type": "Consult" }, - "seconds": 8.94, + "seconds": 3.8, "seed_concurrent": true, "task": "mockmed-05", "verdicts": { @@ -3073,7 +3072,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3092,7 +3091,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 8.94, + "seconds": 3.83, "seed_concurrent": true, "task": "mockmed-06", "verdicts": { @@ -3114,7 +3113,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3133,7 +3132,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 8.98, + "seconds": 3.83, "seed_concurrent": false, "task": "mockmed-07", "verdicts": { @@ -3155,7 +3154,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3174,7 +3173,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 8.96, + "seconds": 3.83, "seed_concurrent": false, "task": "mockmed-08", "verdicts": { @@ -3196,7 +3195,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3215,7 +3214,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 8.93, + "seconds": 3.82, "seed_concurrent": true, "task": "mockmed-09", "verdicts": { @@ -3237,7 +3236,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3256,7 +3255,7 @@ "patient_id": "p3", "type": "Consult" }, - "seconds": 9.02, + "seconds": 3.78, "seed_concurrent": false, "task": "mockmed-10", "verdicts": { @@ -3278,7 +3277,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3297,7 +3296,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 8.94, + "seconds": 3.8, "seed_concurrent": false, "task": "mockmed-11", "verdicts": { @@ -3319,7 +3318,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3338,7 +3337,7 @@ "patient_id": "p2", "type": "Triage" }, - "seconds": 8.92, + "seconds": 3.8, "seed_concurrent": true, "task": "mockmed-12", "verdicts": { @@ -3360,7 +3359,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "CONFIRMED" }, "hand-written+complete": { "error": null, @@ -3379,7 +3378,7 @@ "patient_id": "p2", "type": "Triage" }, - "seconds": 34.86, + "seconds": 34.64, "seed_concurrent": false, "task": "mockmed-13", "verdicts": { @@ -3401,7 +3400,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3420,7 +3419,7 @@ "patient_id": "p3", "type": "Consult" }, - "seconds": 34.74, + "seconds": 34.79, "seed_concurrent": false, "task": "mockmed-14", "verdicts": { @@ -3442,7 +3441,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3461,7 +3460,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 34.79, + "seconds": 34.77, "seed_concurrent": true, "task": "mockmed-15", "verdicts": { @@ -3483,7 +3482,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3502,7 +3501,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 34.86, + "seconds": 34.79, "seed_concurrent": false, "task": "mockmed-16", "verdicts": { @@ -3524,7 +3523,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3543,7 +3542,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 34.84, + "seconds": 34.79, "seed_concurrent": false, "task": "mockmed-17", "verdicts": { @@ -3565,7 +3564,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3584,7 +3583,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 34.81, + "seconds": 34.74, "seed_concurrent": true, "task": "mockmed-18", "verdicts": { @@ -3606,7 +3605,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3625,7 +3624,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 34.3, + "seconds": 34.67, "seed_concurrent": false, "task": "mockmed-19", "verdicts": { @@ -3647,7 +3646,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3666,7 +3665,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 34.79, + "seconds": 34.77, "seed_concurrent": false, "task": "mockmed-20", "verdicts": { @@ -3688,7 +3687,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3707,7 +3706,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 34.76, + "seconds": 34.74, "seed_concurrent": true, "task": "mockmed-21", "verdicts": { @@ -3729,7 +3728,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3748,7 +3747,7 @@ "patient_id": "p3", "type": "Consult" }, - "seconds": 34.81, + "seconds": 34.73, "seed_concurrent": false, "task": "mockmed-22", "verdicts": { @@ -3770,7 +3769,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3811,7 +3810,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3830,7 +3829,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 34.72, + "seconds": 34.74, "seed_concurrent": true, "task": "mockmed-24", "verdicts": { @@ -3852,7 +3851,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3871,7 +3870,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 34.8, + "seconds": 34.78, "seed_concurrent": false, "task": "mockmed-25", "verdicts": { @@ -3893,7 +3892,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3912,7 +3911,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 34.77, + "seconds": 34.73, "seed_concurrent": false, "task": "mockmed-26", "verdicts": { @@ -3934,7 +3933,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3953,7 +3952,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 34.76, + "seconds": 34.86, "seed_concurrent": true, "task": "mockmed-27", "verdicts": { @@ -3975,7 +3974,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -3994,7 +3993,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 50.31, + "seconds": 50.15, "seed_concurrent": true, "task": "mockmed-28", "verdicts": { @@ -4016,7 +4015,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4035,7 +4034,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 50.25, + "seconds": 50.21, "seed_concurrent": true, "task": "mockmed-29", "verdicts": { @@ -4057,7 +4056,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4076,7 +4075,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 50.22, + "seconds": 50.08, "seed_concurrent": true, "task": "mockmed-30", "verdicts": { @@ -4098,7 +4097,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4117,7 +4116,7 @@ "patient_id": "p3", "type": "Triage" }, - "seconds": 34.83, + "seconds": 34.75, "seed_concurrent": false, "task": "mockmed-31", "verdicts": { @@ -4139,7 +4138,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4158,7 +4157,7 @@ "patient_id": "p2", "type": "Consult" }, - "seconds": 34.92, + "seconds": 34.67, "seed_concurrent": false, "task": "mockmed-32", "verdicts": { @@ -4180,7 +4179,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4199,7 +4198,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 34.74, + "seconds": 34.71, "seed_concurrent": true, "task": "mockmed-33", "verdicts": { @@ -4221,7 +4220,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4240,7 +4239,7 @@ "patient_id": "p1", "type": "Consult" }, - "seconds": 34.75, + "seconds": 34.71, "seed_concurrent": false, "task": "mockmed-34", "verdicts": { @@ -4262,7 +4261,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4303,7 +4302,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4322,7 +4321,7 @@ "patient_id": "p1", "type": "Triage" }, - "seconds": 34.7, + "seconds": 34.76, "seed_concurrent": true, "task": "mockmed-36", "verdicts": { @@ -4344,7 +4343,7 @@ }, "hand-written": { "error": null, - "verdict": "INDETERMINATE" + "verdict": "REFUTED" }, "hand-written+complete": { "error": null, @@ -4355,12 +4354,12 @@ ] } }, - "git_head": "cfa11472b7cb656764577b599ddf66874db446bc", + "git_head": "ae4ecc75e9d447df132a9224ce4a05ecd2f513d5", "platform": "macOS-15.7.3-arm64-arm-64bit", "protocol": "PROTOCOL.md", "python": "3.12.13", "schema": "openadapt.zero-config-verify-results/v1", - "started": "2026-11-09T17:28:57Z", + "started": "2026-11-10T00:35:57Z", "verdict": { "zero_config_ready_arms": { "mockloan": [ diff --git a/packages/flow/docs/EFFECT_KIT.md b/packages/flow/docs/EFFECT_KIT.md index 26e316ecf..1e5f082a0 100644 --- a/packages/flow/docs/EFFECT_KIT.md +++ b/packages/flow/docs/EFFECT_KIT.md @@ -1086,7 +1086,7 @@ Per-kind required fields: | `ui_readback` | `ui_readback` section: `record_url`, `sign_in` (env-var names for the read-back account), `ready_selector`, `row_selector`, `fields`, `identity` | `url_params`, `empty_selector`, `not_found_selector`, `read_post_paths`, per-field `attribute` / `date_format` / `casefold`, `truncation_markers`, `navigation_timeout_s` | | `desktop_readback` | `desktop_readback` section: `session` (a `backend:`-style config for the read-back session), `search` (`box`, `query`), `ready`, `identity`, `fields` | `credentials` (env-var names), `sign_in`, `search.submit`, `open.results`, `not_found`, per-control `shows` / `region` / `at`, per-field `date_format` / `casefold`, `min_ocr_confidence`, `navigation_timeout_s`, `settle_s` | | `deferred_report` | `deferred_report` section: `report` (`format`, and for csv/text its columns or pattern and `accept`), `key`, `deadline_hours`, `ledger` | `report.reject`, `describe` | -| `rest` | `base_url` | `records_path` (may contain `{placeholder}`s), `records_key`, `records_next_path`, `records_total_path`, `records_complete`, `records_snapshot_path`, `path_params`, `auth`, `records_drop_fields` | +| `rest` | `base_url` | `records_path` (may contain `{placeholder}`s), `records_key`, `records_next_path`, `records_total_path`, `records_complete`, `records_list_keys`, `records_other_lists`, `records_snapshot_path`, `path_params`, `auth`, `records_drop_fields` | | `graphql` | `base_url` + `graphql_query` | `graphql_variables` (`{param: ...}` entity binding), `graphql_records_path`, `graphql_freshness_field` + `graphql_freshness_window_s`, `auth` | | `fhir` | `base_url` | `resource_type`, `search_params`, `search_param_exprs`, `field_paths`, `access_token_env` or `smart_backend`, `verify_tls`, `records_drop_fields`, `records_snapshot_path` | | `sql` | `sql_query` + (`sqlite_database` or `sql_driver`) | `sql_query_params`, `sql_connect_args`, `sql_password_env`, `records_drop_fields` | @@ -1139,6 +1139,16 @@ whose pages disagree on the total, or whose body omits it is unreadable. Without next-page chain must reach its declared terminal marker; an unpaged read requires a validated `records_complete: true` declaration. +`records_list_keys` names the top-level keys whose value is a list, for a +whole-store read that returns one list per collection. A body without a list +under one of them, or with an object value (which may hold records no +reviewed read sees), is unreadable. A read with `records_other_lists: true` (and +no `records_key`) returns the records of every list under any other key, so a +row written to a collection that appeared after review is seen. `scaffold-verifier +--probe-reads` sets both on a prefix-root read it uses in place of a collection +read (only when every top-level value is a list of objects or a scalar): one read per list it saw (empty lists included), plus one for any other +list, where the save may add no row. + A redirect is a failed read: the configured auth headers never follow it to another host. Why a read was unreadable is in the verdict's reason and the state's `detail.read_problem`, for example diff --git a/packages/flow/openadapt_flow/deployment.py b/packages/flow/openadapt_flow/deployment.py index 831ce3e26..5fcf4f29c 100644 --- a/packages/flow/openadapt_flow/deployment.py +++ b/packages/flow/openadapt_flow/deployment.py @@ -406,6 +406,14 @@ class EffectsConfig(BaseModel): #: number of returned records. Prefer ``records_next_path`` or #: ``records_total_path`` when the endpoint exposes pagination metadata. records_complete: Optional[StrictBool] = None + #: ``rest`` only: the top-level keys whose value is a list, for a + #: whole-store read with one list per collection, as it was reviewed. A + #: body without a list under one of these reads as unreadable. + records_list_keys: Optional[list[StrictStr]] = None + #: ``rest`` only, with ``records_list_keys`` and no ``records_key``: the + #: read returns the records of every list under any OTHER key, so a row + #: written to a collection that appeared after review is seen. + records_other_lists: Optional[StrictBool] = None #: ``rest`` / ``fhir``: response-body path to a token whose reviewed #: endpoint semantics bind every page to one immutable result snapshot. #: A next link or total alone is not a consistent multi-page read. @@ -650,6 +658,8 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: "records_next_path", "records_total_path", "records_complete", + "records_list_keys", + "records_other_lists", "records_snapshot_path", "records_read_context", "records_consistency", @@ -679,6 +689,14 @@ def _validate_records_drop_fields(self) -> "EffectsConfig": ) if self.records_complete is not None and kind != "rest": raise ValueError("effects.records_complete applies only to kind rest") + if self.records_list_keys is not None and kind != "rest": + raise ValueError("effects.records_list_keys applies only to kind rest") + if self.records_other_lists and ( + self.records_list_keys is None or self.records_key is not None + ): + raise ValueError( + "effects.records_other_lists needs records_list_keys and no records_key" + ) if self.records_snapshot_path is not None and kind not in ("rest", "fhir"): raise ValueError( "effects.records_snapshot_path applies only to kinds rest and fhir" @@ -2240,6 +2258,8 @@ def sanitized(value: Any, *, key: str = "") -> Any: "records_next_path", "records_total_path", "records_complete", + "records_list_keys", + "records_other_lists", "records_snapshot_path", "records_read_context", "records_consistency", @@ -2417,6 +2437,10 @@ def _build_effect_verifier_unredacted( snapshot_path=cfg.records_snapshot_path, consistency=cfg.records_consistency, quiescence=quiescence, + list_keys=( + None if cfg.records_list_keys is None else tuple(cfg.records_list_keys) + ), + other_lists=bool(cfg.records_other_lists), ) if kind == "fhir": diff --git a/packages/flow/openadapt_flow/read_discovery.py b/packages/flow/openadapt_flow/read_discovery.py index 78c5d03bb..d3216b25f 100644 --- a/packages/flow/openadapt_flow/read_discovery.py +++ b/packages/flow/openadapt_flow/read_discovery.py @@ -21,6 +21,19 @@ so, lists every probe, and needs a person to name the read. Nothing is invented. +A promoted collection read sees only the written collection, so a write +elsewhere in the store would pass it (an open ``collateral_unaudited`` miss) +and the check is refused. Before that, :func:`cover` looks for a read that +covers the whole save: the sibling collections under the same prefix that the +app itself read (taken from its captured ``GET`` requests, path only), and the +prefix root (``GET ``). The root is promoted in place of the +collection only when it holds the written record, holds every record the +collection and each sibling returned, is one unpaged response, and every +captured write went under the prefix. The check then reads every list the +root returns: the one with the saved record may gain exactly one row, the +others none. When nothing covers the save, the refusal stands and its reason +says what was probed. + Every probe (URL with record ids masked, status, outcome) is recorded in ``search.read_discovery`` and from there in the proof. """ @@ -28,10 +41,11 @@ from __future__ import annotations import ipaddress +import json import urllib.error import urllib.request from dataclasses import dataclass -from typing import Any, Mapping, Optional +from typing import Any, Mapping, Optional, Sequence from urllib.parse import urlsplit from openadapt_flow import record_check_synthesis as rcs @@ -42,6 +56,8 @@ PROBE_TIMEOUT_S = 5.0 #: Most probes for one save. MAX_PROBES = 12 +#: Most probes when looking for a read that covers the whole save. +MAX_COVER_PROBES = 8 _PROBE_METHODS = frozenset({"POST", "PUT", "PATCH"}) _ID_KEYS = ("id", "_id", "uuid") @@ -75,8 +91,18 @@ def http_get(url: str) -> tuple[Optional[int], Optional[str], bool]: """``(status, text, truncated)`` for one GET on the local machine.""" if not is_local_origin(url): - raise ProbeRefused(f"probe refused: {urlsplit(url).netloc} is not local") - opener = urllib.request.build_opener(_NoRedirect) + raise ProbeRefused(f"probe refused: {urlsplit(url).hostname} is not local") + parts = urlsplit(url) + if parts.username is not None or parts.password is not None: + # Never send credentials: drop any user:password from the URL. + host = parts.hostname or "" + if ":" in host: + host = f"[{host}]" + netloc = host + (f":{parts.port}" if parts.port else "") + url = parts._replace(netloc=netloc).geturl() + # No proxy (a proxy would carry the probe off this machine), no cookies, + # no redirects. + opener = urllib.request.build_opener(urllib.request.ProxyHandler({}), _NoRedirect) request = urllib.request.Request( url, method="GET", headers={"Accept": "application/json"} ) @@ -230,7 +256,7 @@ def unavailable(reason: str) -> None: return None if probe_base is not None and not is_local_origin(probe_base): unavailable( - f"the probe base {urlsplit(probe_base).netloc} is not the local " + f"the probe base {urlsplit(probe_base).hostname} is not the local " "machine; probes run only on a local copy of the app" ) return None @@ -249,7 +275,7 @@ def unavailable(reason: str) -> None: continue if not is_local_origin(base): unavailable( - f"the write went to {urlsplit(base).netloc}, not the local " + f"the write went to {urlsplit(base).hostname}, not the local " "machine; give --probe-base with a local copy of the app" ) return None @@ -334,3 +360,378 @@ def unavailable(reason: str) -> None: + "; a person must name the read" ) return None + + +# --------------------------------------------------------------------------- +# A read that covers the whole save +# --------------------------------------------------------------------------- + + +def _under(path: str, prefix: str) -> bool: + return path == prefix or path.startswith(prefix.rstrip("/") + "/") + + +def _holds(records: list[dict[str, Any]], record: Mapping[str, Any]) -> bool: + """True when some record of ``records`` carries every field of ``record``.""" + + items = {k: _json_key(v) for k, v in record.items()} + return any( + all(k in r and _json_key(r[k]) == v for k, v in items.items()) for r in records + ) + + +def _json_key(value: Any) -> str: + return json.dumps(value, sort_keys=True, default=str) + + +def sibling_paths( + captured: Sequence[Mapping[str, Any]], + *, + origin: str, + prefix: str, + written: str, +) -> list[str]: + """Collections under ``prefix`` the app itself read (path only). + + Only ``/`` on the app's own origin, from a successful + ``GET`` the app made, other than the written collection's spellings. The query string is + dropped (it may carry a session value), and a segment that looks like a + record id or holds a session marker is skipped. + """ + + skip = {s.lower() for s in _spellings(written)} + out: list[str] = [] + for line in captured: + if str(line.get("method") or "").upper() != "GET" or not rcs._ok(line): + continue + url = str(line.get("url") or "") + if rcs._origin(url) != origin: + continue + path = urlsplit(url).path.rstrip("/") + if not path.startswith(prefix + "/"): + continue + name = path[len(prefix) + 1 :] + if not name or "/" in name or name.lower() in skip: + continue + if rcs._RECORD_SEGMENT.search(name) or rcs._SESSION_VALUE.search(name): + continue + out.append(f"{prefix}/{name}") + return list(dict.fromkeys(out)) + + +def cover( + params: Mapping[str, str], + search: rcs.ReadSearch, + promoted: rcs.Surface, + *, + captured: Sequence[Mapping[str, Any]], + probe_base: Optional[str] = None, + get: Any = None, +) -> Optional[rcs.Surface]: + """Replace ``promoted`` with a read that covers the whole save, if one is found. + + Returns the new saved-record surface, or ``None`` with + ``search.read_discovery["covering_read"]`` saying why nothing covered it. + Same rules as :func:`discover`: ``GET`` only, the local machine only, no + credentials, no redirects, every probe recorded. + """ + + get = get or http_get + found = search.read_discovery + if found is None: + return None + probes: list[dict[str, Any]] = found.setdefault("probes", []) + covering: dict[str, Any] = {"status": "not found"} + found["covering_read"] = covering + lookup, _ = rcs.param_lookup(params) + path = str(promoted.config.get("records_path") or "") + + def none(reason: str) -> None: + covering["reason"] = reason + + if "{" in path or "?" in path or "/" not in path: + none(f"the probed read {path} is not a plain collection path") + return None + prefix, written = path.rsplit("/", 1) + writes = list(search.write_requests) + outside = [ + line + for line in writes + if not _under(urlsplit(str(line.get("url") or "")).path, prefix or "/") + ] + if outside: + none( + "a captured write went outside " + + (prefix or "/") + + ", so no read under it can see the whole save" + ) + return None + origins = {rcs._origin(str(line.get("url") or "")) for line in writes} + base = str(promoted.config.get("base_url") or "").rstrip("/") + if probe_base is not None: + base = probe_base.rstrip("/") + if not is_local_origin(base): + none(f"{urlsplit(base).hostname} is not the local machine") + return None + sent = _sent_params(writes[0], params) if writes else {} + for line in writes[1:]: + sent.update(_sent_params(line, params)) + + budget = [MAX_COVER_PROBES] + + def probe(cand_path: str, kind: str) -> tuple[dict[str, Any], Any, str]: + url = base + cand_path + safe = rcs._templated(url, lookup).safe + row: dict[str, Any] = { + "method": "GET", + "path": safe, + "kind": kind, + "purpose": "a read that covers the whole save", + } + probes.append(row) + budget[0] -= 1 + status, text, truncated = get(url) + row["status"] = status + if status is None or not 200 <= status < 300 or text is None: + row["outcome"] = "no read here" + return row, None, url + fake = { + "method": "GET", + "url": url, + "status": status, + "response_text": text, + "response_truncated": truncated, + "t": None, + } + body = rcs._json_body(fake) + if body is None: + row["outcome"] = "not JSON (or cut)" + return row, None, url + return row, fake, url + + # 1. The collections the app read beside the written one. + seen: dict[str, list[dict[str, Any]]] = {path: [dict(r) for r in promoted.after]} + siblings: list[str] = [] + for origin in sorted(origins): + siblings += sibling_paths( + captured, origin=origin, prefix=prefix, written=written + ) + siblings = list(dict.fromkeys(siblings)) + unread: list[str] = [] + for sib in siblings: + if budget[0] <= 0: + unread.append(sib) + continue + row, fake, _ = probe(sib, "sibling") + if fake is None: + unread.append(sib) + continue + records = [ + dict(r) + for listing in rcs._listings(rcs._json_body(fake)) + for r in listing.resources + ] + seen[sib] = records + row["outcome"] = f"read: {len(records)} record(s); the root must hold them all" + if unread: + none( + "the app read " + + ", ".join(unread) + + " beside the written collection, but a probe could not read it" + ) + return None + + # 2. The prefix root: one read of every collection under the prefix. + roots = list(dict.fromkeys([prefix or "/", (prefix or "") + "/"])) + why: list[str] = [] + for root in roots: + if budget[0] <= 0: + why.append(f"GET {root}: not probed (probe limit)") + break + row, fake, url = probe(root, "root") + if fake is None: + why.append(f"GET {row['path']}: {row['outcome']} (status {row['status']})") + continue + body = rcs._json_body(fake) + listings = rcs._listings(body) + if not listings: + row["outcome"] = "holds no record list" + why.append(f"GET {row['path']}: holds no record list") + continue + taken = set(search.taken) - {promoted.name, promoted.content_name} + target: Optional[rcs.Surface] = None + target_listing = None + for listing in listings: + surface, _why = rcs._surface_from_listing( + fake, + listing, + lookup, + before_line=None, + taken=set(taken), + excluded_values=params.values(), + ) + if surface is None or not set(sent) <= set(surface.bound.values()): + continue + if target is None or len(surface.bound) > len(target.bound): + target, target_listing = surface, listing + if target is None or target_listing is None: + row["outcome"] = "no record holds every written value" + why.append(f"GET {row['path']}: does not hold the written record") + continue + everything = [dict(r) for listing in listings for r in listing.resources] + missing = [ + name + for name, records in seen.items() + if not all(_holds(everything, r) for r in records) + ] + if missing: + row["outcome"] = "does not hold every record of " + ", ".join(missing) + why.append( + f"GET {row['path']}: does not hold every record of " + + ", ".join(missing) + + ", so it is not the whole store" + ) + continue + if not target.observed_complete: + row["outcome"] = "not one whole response: " + "; ".join(target.incomplete) + why.append(f"GET {row['path']}: {row['outcome']}") + continue + if any(listing.kind != "rest" for listing in listings): + row["outcome"] = "holds a list the REST check can't read" + why.append(f"GET {row['path']}: {row['outcome']}") + continue + # Fail closed on shape: every top-level value must be a list of + # objects (a collection) or a scalar. An object may hold records the + # check would not read (a nested collection, an id-keyed map). + nested = ( + sorted( + str(k) + for k, v in body.items() + if not isinstance(v, list) + and not (v is None or isinstance(v, (str, int, float, bool))) + ) + if isinstance(body, dict) + else [] + ) + if nested: + row["outcome"] = ( + "root holds a non-list value that may contain records: " + + ", ".join(nested) + ) + why.append(f"GET {row['path']}: {row['outcome']}") + continue + if isinstance(body, list) and any(not isinstance(i, dict) for i in body): + row["outcome"] = "holds a list of non-records" + why.append(f"GET {row['path']}: {row['outcome']}") + continue + # Every top-level list is a collection the check must read, an empty + # one included (``_listings`` skips lists that hold no object). + lists: dict[str, list[Any]] = ( + {str(k): v for k, v in body.items() if isinstance(v, list)} + if isinstance(body, dict) + else {} + ) + odd = sorted( + k for k, v in lists.items() if any(not isinstance(i, dict) for i in v) + ) + if odd: + row["outcome"] = "holds a list of non-records under " + ", ".join(odd) + why.append(f"GET {row['path']}: {row['outcome']}") + continue + # Promote: the root replaces the collection; every list it returns + # is read (the saved record's may gain one row, the others none), and + # a list under any other key makes the read unreadable. + safe = row["path"] + taken.update({target.name, target.content_name}) + target.source = f"probe GET {safe} (read-only, after the save)" + target.store_wide = True + if isinstance(body, dict): + target.config["records_list_keys"] = sorted(lists) + guarded: list[rcs.Surface] = [] + for key in sorted(lists): + if key == target_listing.records_key: + continue + records = [dict(r) for r in lists[key]] + config: dict[str, Any] = { + "kind": "rest", + "base_url": target.config["base_url"], + "records_path": target.config["records_path"], + "records_key": key, + "records_list_keys": sorted(lists), + } + guarded.append( + rcs.Surface( + name=rcs._unique_name(f"{target.name}_{key}", taken), + config=config, + after=records, + before=[dict(r) for r in records], + before_recorded=False, + source=f"probe GET {safe} (read-only, after the save), list {key!r}", + observed_complete=target.observed_complete, + incomplete=[], + ) + ) + if isinstance(body, dict): + # Any list under a key the probe did not see: a collection that + # appeared later. The save may add no row there. + guarded.append( + rcs.Surface( + name=rcs._unique_name(f"{target.name}_other_lists", taken), + config={ + "kind": "rest", + "base_url": target.config["base_url"], + "records_path": target.config["records_path"], + "records_key": None, + "records_list_keys": sorted(lists), + "records_other_lists": True, + }, + after=[], + before=[], + before_recorded=False, + source=f"probe GET {safe} (read-only, after the save), every " + "list not seen when probed", + observed_complete=target.observed_complete, + incomplete=[], + ) + ) + search.surfaces = [s for s in search.surfaces if s is not promoted] + search.surfaces += [target] + guarded + search.taken = taken + search.closed_world = True + search.closed_world_basis = "probed" + row["outcome"] = ( + "confirmed: holds the written record and every record of " + ", ".join(seen) + ) + row["confirmed_fields"] = sorted(target.bound) + covering.update( + status="promoted", + read=safe, + lists=sorted(lists) if isinstance(body, dict) else [None], + basis=( + f"GET {safe} is the root of every captured write; it holds the " + "written record and every record of " + + ", ".join(seen) + + " in one unpaged response" + ), + ) + found.update( + read=safe, + surface=target.name, + collection_read=found.get("read"), + needs_operator_confirmation=True, + ) + search.notes.append( + f"read discovery: GET {safe} covers the whole save (it holds every " + f"record of {', '.join(seen)}); the check reads it instead of " + f"{path}. A person confirms it is the whole store" + ) + return target + none( + "; ".join(why) + + ( + "; the app read no other collection under " + (prefix or "/") + if not siblings + else "" + ) + ) + return None diff --git a/packages/flow/openadapt_flow/record_check_synthesis.py b/packages/flow/openadapt_flow/record_check_synthesis.py index 4d3c23ccb..84444a68b 100644 --- a/packages/flow/openadapt_flow/record_check_synthesis.py +++ b/packages/flow/openadapt_flow/record_check_synthesis.py @@ -213,6 +213,10 @@ class Surface: complete_basis: Optional[str] = None #: The request or response says it is one page (see :func:`_paging_signs`). paged: bool = False + #: The read returns the whole store, not one collection (a covering read + #: found by :func:`openadapt_flow.read_discovery.cover`): a row written + #: anywhere in the store shows up here, so the save may add exactly one. + store_wide: bool = False @property def is_target(self) -> bool: @@ -1399,6 +1403,17 @@ def effects_for( confirmed=confirmed, ) ) + if surface.store_wide: + effects.append( + _effect( + surface.name, + "the save adds exactly one row to the whole store read", + match={}, + expected_count=1, + forbid_collateral_loss=False, + confirmed=confirmed, + ) + ) else: effects.append( _effect( @@ -1716,6 +1731,28 @@ def other_records( f"saved twice in {surface.name}", {**base, surface.name: (before, after + [copy])}, ) + if surface.store_wide: + # The read is the whole store: a stray row written to another + # collection lands here, with values unlike the saved record's. + elsewhere = dict(target) + for fld in list(surface.bound) + list(surface.constants): + elsewhere[fld] = f"stray {elsewhere.get(fld)}" + if "id" in elsewhere or surface.has_ids: + elsewhere["id"] = _fresh_id(after + before) + # Only a bare-list root is the whole store in one list; on an + # object root this is an unrelated row in the saved record's + # own collection (other collections have their own reads). + whole = surface.config.get("records_key") is None + run( + "collateral_unaudited" if whole else "unrelated_row", + ( + f"a stray row written elsewhere in the store (it shows in " + f"{surface.name})" + if whole + else f"an unrelated row added to {surface.name}" + ), + {**base, surface.name: (before, after + [elsewhere])}, + ) identity = [ f for f, p in surface.bound.items() if _identity_like(params.get(p, "")) @@ -1774,7 +1811,12 @@ def other_records( stray["_stray"] = "1" run( "collateral_unaudited", - f"a stray row written to {surface.name}", + ( + f"a stray row written to a collection {surface.name} did not " + "show when probed (a new list in the store read)" + if surface.config.get("records_other_lists") + else f"a stray row written to {surface.name}" + ), {**base, surface.name: (before, after + [stray])}, ) @@ -1992,9 +2034,14 @@ def synthesize( if db_path is not None: reads_from_sqlite(Path(db_path), params, search) if probe_reads and not any(s.is_target for s in search.surfaces): - from openadapt_flow.read_discovery import discover - - discover(params, search, probe_base=probe_base) + from openadapt_flow.read_discovery import cover, discover + + promoted = discover(params, search, probe_base=probe_base) + if promoted is not None and not complete_read and not search.closed_world: + # The probed read is the written collection only. Before the + # refusal below, look for a read that covers the whole save. + captured = load_network(Path(recording_dir)) if recording_dir else [] + cover(params, search, promoted, captured=captured, probe_base=probe_base) if not any(s.is_target for s in search.surfaces): return Proposal(params, search, [], {}, []) if rest_setups is not None: @@ -2062,14 +2109,27 @@ def _refuse_partial_probed_read(proposal: Proposal) -> None: ] + [f"not covered: {item}" for item in proposal.search.uncovered] if not unaudited: return + covering = found.get("covering_read") or {} + searched = ( + f" No read covering the whole save was found: {covering['reason']}." + if covering.get("status") == "not found" and covering.get("reason") + else "" + ) + scope = ( + "covers only the written collection" + if not covering.get("status") == "promoted" + else "does not cover the whole save" + ) found.update( status="refused", needs_operator_confirmation=True, reason=( - f"the probed read {found.get('read')} covers only the written " - "collection, and a write outside it would pass the check (" + f"the probed read {found.get('read')} {scope}, and a write outside " + "it would pass the check (" + "; ".join(unaudited) - + "). No runnable check was written; confirm the read covers " + + ")." + + searched + + " No runnable check was written; confirm the read covers " "everything the save can change (--complete-read), give --db, or " "name a fuller read" ), @@ -2091,6 +2151,7 @@ def _refuse_partial_probed_read(proposal: Proposal) -> None: "changed": "unrelated_edit", "deleted": "unrelated_edit", "collateral_unaudited": "unrelated_edit", + "unrelated_row": "unrelated_edit", } diff --git a/packages/flow/openadapt_flow/runtime/effects/rest.py b/packages/flow/openadapt_flow/runtime/effects/rest.py index e4ef59e32..79995345c 100644 --- a/packages/flow/openadapt_flow/runtime/effects/rest.py +++ b/packages/flow/openadapt_flow/runtime/effects/rest.py @@ -122,6 +122,15 @@ class RestRecordVerifier(VerifierAdapterBase): that one response contains the complete matching record set. This is for validated, unpaged endpoints; it is never inferred from response length. + list_keys: The top-level keys whose value is a list in a whole-store + read that holds one list per collection, as the read was reviewed. + Set, a body that is not a JSON object, lacks a list under one of + these keys, or holds an object value (which may contain records no + reviewed read sees), reads as unreadable. + other_lists: With ``list_keys`` (and ``records_key=None``): the read + returns the records of every list under a key NOT in + ``list_keys``. A collection that appears after review is read + here, so a row written to it is seen instead of passing unread. snapshot_path: Dotted body path to the immutable view identifier in a reviewed endpoint contract. Multi-page reads require the same exact token on every page; its observed stability alone does not @@ -151,8 +160,14 @@ def __init__( snapshot_path: Optional[str] = None, consistency: Optional[str] = None, quiescence: Any = None, + list_keys: Optional[tuple[str, ...]] = None, + other_lists: bool = False, ) -> None: self.base_url = base_url.rstrip("/") + self.list_keys = None if list_keys is None else frozenset(list_keys) + self.other_lists = bool(other_lists) + if self.other_lists and (self.list_keys is None or records_key is not None): + raise ValueError("other_lists needs list_keys and no records_key") self.next_path = next_path self.total_path = total_path self.records_complete = records_complete @@ -237,6 +252,15 @@ def _get_json(self, url: str) -> Any: return _UNREADABLE def _records_of(self, body: Any) -> Optional[list[Any]]: + if self.other_lists: + if not isinstance(body, dict): + return None + return [ + item + for key in sorted(body, key=str) + if key not in (self.list_keys or ()) and isinstance(body[key], list) + for item in body[key] + ] if self.records_key is None: records = body elif isinstance(body, dict): @@ -321,6 +345,30 @@ def _fetch_pass( self.readback_complete = False return None snapshot = current_snapshot + if self.list_keys is not None: + missing = ( + sorted( + k for k in self.list_keys if not isinstance(body.get(k), list) + ) + if isinstance(body, dict) + else ["(the body is not an object)"] + ) + if missing: + self.read_problem = ( + f"GET {url}: the reviewed lists are missing: " + + ", ".join(missing) + ) + self.readback_complete = False + return None + nested = sorted(str(k) for k, v in body.items() if isinstance(v, dict)) + if nested: + # An object may hold records no reviewed read sees. + self.read_problem = ( + f"GET {url}: the body holds an object that may contain " + "records: " + ", ".join(nested) + ) + self.readback_complete = False + return None records = self._records_of(body) if records is None: self.read_problem = f"GET {url}: no records list in the body" diff --git a/packages/flow/openadapt_flow/scaffold_verifier.py b/packages/flow/openadapt_flow/scaffold_verifier.py index c11a81276..a73a1a496 100644 --- a/packages/flow/openadapt_flow/scaffold_verifier.py +++ b/packages/flow/openadapt_flow/scaffold_verifier.py @@ -805,6 +805,14 @@ def build_record_check( proposals, destination ) check, proof = rcs.write(proposal, destination) + if proposal.search.read_discovery is not None: + import json + + # Every probe, also when one was promoted (the proof holds it too). + _sealed.write_text( + destination / READ_DISCOVERY_FILE, + json.dumps(proposal.search.read_discovery, indent=2, sort_keys=True) + "\n", + ) return RecordCheckResult( working=proposal.working, check_path=check, diff --git a/packages/flow/public-artifacts.json b/packages/flow/public-artifacts.json index 7fa0470c4..dfa17c208 100644 --- a/packages/flow/public-artifacts.json +++ b/packages/flow/public-artifacts.json @@ -552,6 +552,7 @@ "benchmark/zero_config_verify/generated/mockmed.Z-net.yaml", "benchmark/zero_config_verify/results-run1.json", "benchmark/zero_config_verify/results-run2.json", + "benchmark/zero_config_verify/results-run3.json", "benchmark/zero_config_verify/results.json", "chrome_extension/background.js", "chrome_extension/content.js", diff --git a/packages/flow/tests/test_read_discovery_cover.py b/packages/flow/tests/test_read_discovery_cover.py new file mode 100644 index 000000000..4b4695b1c --- /dev/null +++ b/packages/flow/tests/test_read_discovery_cover.py @@ -0,0 +1,541 @@ +"""``--probe-reads``: look for a read that covers the whole save. + +A probed collection read sees only the written collection, so a write to +another collection would pass it and the check is refused. Before refusing, +the generator probes the sibling collections the app itself read and the +prefix root, and promotes the root only when it holds the written record and +every record the collection and the siblings returned. + +Synthetic local apps and made-up values only; no model calls. +""" + +from __future__ import annotations + +import json +import re +import threading +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from typing import Any, Optional +from urllib.parse import urlsplit + +import pytest +import requests +import yaml + +from openadapt_flow import read_discovery +from openadapt_flow.record_check_synthesis import render, synthesize +from openadapt_flow.scaffold_verifier import READ_DISCOVERY_FILE, build_record_check + +from ._record_check_fixtures import Capture + +NOTE = "Synthetic memo: release after the second review" +PARAMS = {"account": "acct-1001", "memo": NOTE} +_FLAG = re.compile(r"^\s*needs_operator_confirmation:\s*true\s*$", re.M) + + +class StoreApp: + """A local JSON app with two collections: ``transfers`` and ``fees``. + + ``POST /api/transfer`` saves a transfer; with ``stray`` it also writes a + fee (a wrong save the screen can't see). ``root`` says what ``GET /api`` + returns: ``"split"`` (one list per collection), ``"merged"`` (one list of + every record), ``"partial"`` (transfers only), ``"redirect"`` (a 302 to a + route that holds everything) or ``None`` (404). ``fees_route`` serves + ``GET /api/fees``. Every request is logged with its headers. + """ + + def __init__(self, *, root: Optional[str], fees_route: bool = True) -> None: + self.transfers: list[dict[str, Any]] = [ + {"id": 1, "account": "acct-0001", "memo": "earlier memo"} + ] + self.fees: list[dict[str, Any]] = [ + {"id": 501, "account": "acct-0001", "fee": "2.50"} + ] + self.stray = False + self.root = root + self.fees_route = fees_route + self.log: list[tuple[str, str, dict[str, str]]] = [] + app = self + + class Handler(BaseHTTPRequestHandler): + def log_message(self, *args: Any) -> None: + return None + + def _send(self, status: int, body: Any, **headers: str) -> None: + raw = json.dumps(body).encode() + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(raw))) + for key, value in headers.items(): + self.send_header(key, value) + self.end_headers() + self.wfile.write(raw) + + def _everything(self) -> list[dict[str, Any]]: + return [dict(r, kind="transfer") for r in app.transfers] + [ + dict(r, kind="fee") for r in app.fees + ] + + def do_GET(self) -> None: # noqa: N802 + path = urlsplit(self.path).path + app.log.append(("GET", path, dict(self.headers.items()))) + if path == "/api/transfers": + self._send(200, {"records": app.transfers}) + elif path == "/api/fees" and app.fees_route: + self._send(200, {"records": app.fees}) + elif path in ("/api", "/api/") and app.root == "split": + self._send(200, {"transfers": app.transfers, "fees": app.fees}) + elif path in ("/api", "/api/") and app.root == "omit_empty": + # A collection with no rows is left out of the body. + body = {"transfers": app.transfers} + if app.fees: + body["fees"] = app.fees + self._send(200, body) + elif path in ("/api", "/api/") and app.root == "nested": + self._send( + 200, {"transfers": app.transfers, "billing": {"fees": app.fees}} + ) + elif path in ("/api", "/api/") and app.root == "idmap": + fees = {str(r["id"]): r for r in app.fees} + self._send(200, {"transfers": app.transfers, "fees": fees}) + elif path in ("/api", "/api/") and app.root == "merged": + self._send(200, {"records": self._everything()}) + elif path in ("/api", "/api/") and app.root == "partial": + self._send(200, {"records": app.transfers}) + elif path in ("/api", "/api/") and app.root == "redirect": + self._send(302, {}, Location="/hidden/everything") + elif path == "/hidden/everything": + self._send(200, {"records": self._everything()}) + else: + self._send(404, {"error": "not found"}) + + def do_POST(self) -> None: # noqa: N802 + path = urlsplit(self.path).path + app.log.append(("POST", path, dict(self.headers.items()))) + if path != "/api/transfer": + self._send(404, {"error": "not found"}) + return + length = int(self.headers.get("Content-Length") or 0) + body = json.loads(self.rfile.read(length) or b"{}") + rec = {"id": max(r["id"] for r in app.transfers) + 1, **body} + app.transfers.append(rec) + if app.stray: + app.fees.append( + { + "id": max([r["id"] for r in app.fees], default=500) + 1, + "account": body.get("account"), + "fee": "9.99", + } + ) + self._send(200, {"ok": True, "id": rec["id"]}) + + self.server = ThreadingHTTPServer(("127.0.0.1", 0), Handler) + self.url = f"http://127.0.0.1:{self.server.server_address[1]}" + threading.Thread(target=self.server.serve_forever, daemon=True).start() + + def gets(self) -> list[str]: + return [path for method, path, _ in self.log if method == "GET"] + + def stop(self) -> None: + self.server.shutdown() + self.server.server_close() + + +def _recording(rec: Path, app: StoreApp, *, read_fees: bool = False) -> Path: + """One save through ``app``. With ``read_fees`` the app also lists fees + before the save, with a cookie and a token in the query string.""" + + rec.mkdir(parents=True, exist_ok=True) + cap = Capture() + if read_fees: + cap.get(app.url + "/api/fees?token=synthetic-token", 1.0) + cap.lines[-1]["request_headers"] = {"Cookie": "sid=synthetic-session"} + cap.post_json(app.url + "/api/transfer", dict(PARAMS), 1.9) + for line in cap.lines: + line["frame_url"] = app.url + "/form" + events = [ + {"i": 0, "kind": "type", "text": PARAMS["memo"], "t": 1.2}, + { + "i": 1, + "kind": "click", + "t": 2.0, + "text": "Authorize Release", + "structural": {"role": "button", "name": "Authorize Release"}, + }, + ] + (rec / "events.jsonl").write_text( + "".join(json.dumps(e) + "\n" for e in events), encoding="utf-8" + ) + (rec / "meta.json").write_text(json.dumps({"params": PARAMS}), encoding="utf-8") + cap.write(rec) + return rec + + +def _judge(proposal: Any, app: StoreApp, body: dict[str, str]) -> str: + from openadapt_flow.rl_env.reward import EpisodeJudge + from openadapt_flow.verify_run import parse_verify_config + + text = _FLAG.sub("", render(proposal)) + cfg = parse_verify_config(yaml.safe_load(text), source="generated") + judge = EpisodeJudge(cfg, dict(body), episode_id="cover-test") + try: + requests.post(app.url + "/api/transfer", json=body, timeout=5) + return judge.judge("success", feasible=True).verdict + finally: + judge.close() + + +@pytest.fixture +def make_app(): + apps: list[StoreApp] = [] + + def make(**kwargs: Any) -> StoreApp: + app = StoreApp(**kwargs) + apps.append(app) + return app + + yield make + for app in apps: + app.stop() + + +# --------------------------------------------------------------------------- +# Promotion +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("root", ["split", "merged"]) +def test_root_that_covers_the_save_is_promoted_and_catches_a_stray_row( + tmp_path, make_app, root +): + app = make_app(root=root) + rec = _recording(tmp_path / "rec", app, read_fees=True) + before = (list(app.transfers), list(app.fees)) + app.log.clear() + proposal = synthesize(rec, probe_reads=True) + + assert not proposal.refused + assert proposal.working, proposal.misses + found = proposal.search.read_discovery + assert found["status"] == "promoted" + assert found["read"] == "/api" + assert found["collection_read"] == "/api/transfers" + assert found["covering_read"]["status"] == "promoted" + assert found["needs_operator_confirmation"] is True + assert proposal.search.closed_world_basis == "probed" + paths = [p["path"] for p in found["probes"]] + assert paths[:2] == ["/api/transfer", "/api/transfers"] + assert "/api/fees" in paths and "/api" in paths + # Read-only and recorded: only GETs reached the app, nothing changed. + assert {method for method, _, _ in app.log} == {"GET"} + assert (app.transfers, app.fees) == before + assert len(app.gets()) == len(found["probes"]) + + target = next(s for s in proposal.search.surfaces if s.is_target) + assert target.store_wide and target.config["records_path"] == "/api" + faults = {(r["fault"], r["result"]) for r in proposal.proof()["faults"]} + assert ("collateral_unaudited", "MISSED") not in faults + assert ("collateral_unaudited", "caught") in faults + assert proposal.proof()["closed_world"] == "probed" + + good = {"account": "acct-3003", "memo": "Synthetic memo: covered run"} + assert _judge(proposal, app, good) == "CONFIRMED" + app.stray = True + bad = {"account": "acct-4004", "memo": "Synthetic memo: stray fee run"} + assert _judge(proposal, app, bad) == "REFUTED" + + +@pytest.mark.parametrize("root", ["split", "omit_empty"]) +def test_a_stray_row_in_an_empty_or_absent_collection_is_refuted( + tmp_path, make_app, root +): + """Regression: an empty sibling list (``"fees": []``) or a collection the + root left out while empty must still be read, so a stray fee written + there after the probe is REFUTED, not CONFIRMED.""" + + app = make_app(root=root) + app.fees = [] + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True) + assert proposal.working, proposal.misses + found = proposal.search.read_discovery + assert found["read"] == "/api" + names = {s.config.get("records_key") for s in proposal.search.surfaces} + if root == "split": + assert "fees" in names + other = [s for s in proposal.search.surfaces if s.config.get("records_other_lists")] + assert len(other) == 1 + details = [ + r["detail"] for r in proposal.proof()["faults"] if r["result"] == "caught" + ] + assert any("did not show when probed" in d for d in details) + if root == "split": + assert any(d.endswith("_fees") for d in details) + + good = {"account": "acct-3003", "memo": "Synthetic memo: covered run"} + assert _judge(proposal, app, good) == "CONFIRMED" + app.stray = True + bad = {"account": "acct-4004", "memo": "Synthetic memo: stray fee run"} + assert _judge(proposal, app, bad) == "REFUTED" + + +def test_a_reviewed_list_that_disappears_reads_as_unreadable(make_app): + from openadapt_flow.runtime.effects.rest import RestRecordVerifier + + app = make_app(root="omit_empty") + reader = RestRecordVerifier( + app.url, + records_path="/api", + records_key="transfers", + records_complete=True, + list_keys=("fees", "transfers"), + ) + app.fees = [] + assert reader._fetch_records() is None + assert "fees" in (reader.read_problem or "") + app.fees = [{"id": 501, "account": "acct-0001", "fee": "2.50"}] + assert reader._fetch_records() is not None + + +@pytest.mark.parametrize("seeded", [False, True]) +@pytest.mark.parametrize("root", ["nested", "idmap"]) +def test_a_root_holding_an_object_is_refused(tmp_path, make_app, root, seeded): + """Regression: a collection nested in an object, or an id-keyed map, is a + shape the check can't guard, so the root is never promoted (fail closed + on shape) and no check exists that could CONFIRM a stray fee.""" + + app = make_app(root=root) + if not seeded: + app.fees = [] + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused and not proposal.working + found = proposal.search.read_discovery + assert found["read"] == "/api/transfers" + assert found["covering_read"]["status"] == "not found" + key = "billing" if root == "nested" else "fees" + assert ( + f"root holds a non-list value that may contain records: {key}" + in (found["reason"]) + ) + assert not any(s.store_wide for s in proposal.search.surfaces) + assert not any( + r["fault"] == "collateral_unaudited" and r["result"] == "caught" + for r in proposal.proof()["faults"] + ) + out = tmp_path / "out" + assert build_record_check(rec, out, probe_reads=True) is None + assert not (out / "record_check.proposed.yaml").exists() + + +@pytest.mark.parametrize("later", ["nested", "idmap"]) +def test_an_object_that_appears_after_promotion_is_never_confirmed( + tmp_path, make_app, later +): + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True) + assert proposal.working, proposal.misses + app.root = later + app.stray = True + bad = {"account": "acct-4004", "memo": "Synthetic memo: stray fee run"} + assert _judge(proposal, app, bad) != "CONFIRMED" + + +def test_a_list_of_non_records_in_the_root_is_refused(tmp_path, make_app): + app = make_app(root="split") + app.fees = ["not a record"] # type: ignore[list-item] + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + assert ( + "holds a list of non-records under fees" + in (proposal.search.read_discovery["reason"]) + ) + + +def test_promoted_covering_read_is_written_to_read_discovery_json(tmp_path, make_app): + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app, read_fees=True) + out = tmp_path / "out" + result = build_record_check(rec, out, probe_reads=True) + assert result is not None and result.working + saved = json.loads((out / READ_DISCOVERY_FILE).read_text()) + assert saved["status"] == "promoted" and saved["read"] == "/api" + proof = json.loads((out / "record_check.proof.json").read_text()) + assert proof["read_discovery"]["covering_read"]["status"] == "promoted" + + +# --------------------------------------------------------------------------- +# Refusal +# --------------------------------------------------------------------------- + + +def test_no_root_read_keeps_the_refusal_with_the_reason(tmp_path, make_app): + app = make_app(root=None) + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused and not proposal.working + found = proposal.search.read_discovery + assert found["status"] == "refused" + assert found["covering_read"]["status"] == "not found" + assert "collateral_unaudited" in found["reason"] + assert "No read covering the whole save was found" in found["reason"] + assert "GET /api: no read here (status 404)" in found["reason"] + assert "the app read no other collection under /api" in found["reason"] + roots = [p for p in found["probes"] if p.get("kind") == "root"] + assert [p["path"] for p in roots] == ["/api", "/api/"] + + +def test_a_sibling_alone_covers_only_part_of_the_save_and_refuses(tmp_path, make_app): + """The app read ``/api/fees`` too, but nothing shows the whole store: the + two collections together still leave every other table unread.""" + + app = make_app(root=None) + rec = _recording(tmp_path / "rec", app, read_fees=True) + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + found = proposal.search.read_discovery + assert any( + p["path"] == "/api/fees" and p["kind"] == "sibling" for p in found["probes"] + ) + assert found["covering_read"]["status"] == "not found" + assert "collateral_unaudited" in found["reason"] + + +def test_a_root_without_a_siblings_records_is_not_the_whole_store(tmp_path, make_app): + """``GET /api`` holds the written record but not the fees the app read: + it covers only part of the save, so the refusal stands.""" + + app = make_app(root="partial") + rec = _recording(tmp_path / "rec", app, read_fees=True) + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + found = proposal.search.read_discovery + assert "does not hold every record of /api/fees" in found["reason"] + assert not any(s.store_wide for s in proposal.search.surfaces) + + +def test_an_unreadable_sibling_refuses(tmp_path, make_app): + """The app read ``/api/fees`` while recording, but the probe can't: the + root can't be shown to hold its records, so nothing is promoted.""" + + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app, read_fees=True) + app.fees_route = False + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + assert "could not read it" in proposal.search.read_discovery["reason"] + + +def test_a_write_outside_the_prefix_refuses(tmp_path, make_app): + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app) + lines = (rec / "network.jsonl").read_text().splitlines() + extra = json.loads(lines[-1]) + extra["url"] = app.url + "/audit/entry" + extra["t"] = 1.95 + (rec / "network.jsonl").write_text("\n".join(lines + [json.dumps(extra)]) + "\n") + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + assert "went outside /api" in proposal.search.read_discovery["reason"] + + +def test_declared_complete_read_skips_the_covering_search(tmp_path, make_app): + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app) + proposal = synthesize(rec, probe_reads=True, complete_read=True) + found = proposal.search.read_discovery + assert "covering_read" not in found + assert found["read"] == "/api/transfers" + + +# --------------------------------------------------------------------------- +# Probe rules: no credentials, no redirects, GET on this machine only +# --------------------------------------------------------------------------- + + +def test_probes_send_no_credentials_and_drop_the_query(tmp_path, make_app): + app = make_app(root="split") + rec = _recording(tmp_path / "rec", app, read_fees=True) + app.log.clear() + synthesize(rec, probe_reads=True) + assert app.log + for method, path, headers in app.log: + assert method == "GET" + names = {k.lower() for k in headers} + assert not names & {"cookie", "authorization", "proxy-authorization"} + assert "token" not in path + found = synthesize(rec, probe_reads=True).search.read_discovery + assert "synthetic-token" not in json.dumps(found) + assert "synthetic-session" not in json.dumps(found) + + +def test_http_get_strips_credentials_from_the_url(make_app): + app = make_app(root="split") + host = urlsplit(app.url) + status, text, _ = read_discovery.http_get( + f"http://user:secret@{host.hostname}:{host.port}/api/transfers" + ) + assert status == 200 and text + [(_, _, headers)] = app.log + assert not {k.lower() for k in headers} & {"authorization", "cookie"} + + +def test_redirects_are_not_followed(tmp_path, make_app): + app = make_app(root="redirect") + rec = _recording(tmp_path / "rec", app, read_fees=True) + app.log.clear() + proposal = synthesize(rec, probe_reads=True) + assert proposal.refused + assert "/hidden/everything" not in app.gets() + roots = [p for p in proposal.search.read_discovery["probes"] if p["kind"] == "root"] + assert roots and all(p["status"] == 302 for p in roots) + assert all(p["outcome"] == "no read here" for p in roots) + + +def test_cover_never_probes_a_remote_base(make_app): + from openadapt_flow import record_check_synthesis as rcs + + sent: list[str] = [] + search = rcs.ReadSearch() + search.read_discovery = {"probes": []} + search.write_requests = [ + {"method": "POST", "url": "https://app.example/api/transfer"} + ] + promoted = rcs.Surface( + name="api_transfers", + config={"base_url": "https://app.example", "records_path": "/api/transfers"}, + after=[], + before=[], + before_recorded=False, + source="probe", + ) + assert ( + read_discovery.cover( + PARAMS, search, promoted, captured=[], get=lambda url: sent.append(url) + ) + is None + ) + assert sent == [] + assert "not the local machine" in search.read_discovery["covering_read"]["reason"] + + +def test_sibling_paths_come_from_the_apps_own_gets(): + origin = "http://127.0.0.1:9" + captured = [ + {"method": "GET", "url": origin + "/api/fees?token=x"}, + {"method": "GET", "url": origin + "/api/transfers"}, + {"method": "GET", "url": origin + "/api/fees/12345"}, + {"method": "GET", "url": "http://127.0.0.1:10/api/ledger"}, + {"method": "POST", "url": origin + "/api/notes"}, + {"method": "GET", "url": origin + "/static/app.js"}, + {"method": "GET", "url": origin + "/api/accounts", "status": 200}, + {"method": "GET", "url": origin + "/api/missing", "status": 404}, + ] + for line in captured: + line.setdefault("status", 200) + assert read_discovery.sibling_paths( + captured, origin=origin, prefix="/api", written="transfers" + ) == ["/api/fees", "/api/accounts"]