benchmark: API-step generalization results on seven apps

packages/flow/benchmark/api_generalization/README.md

@@ -0,0 +1,100 @@
+# API step generalization: results
+
+Does the API step, which repeats a recorded save as the request the app's own
+page sends, work beyond the OpenEMR referral? This experiment ran ten save
+workflows on seven open-source apps, each in a local container with
+synthetic data, and judged every run against the app's own database
+([PROTOCOL.md](PROTOCOL.md)). The engine changes it depends on are in the
+API-step shapes PR (forms) and the JSON/GraphQL PR.
+
+## Result
+
+- **Safety.** Across 434 runs of every case (normal, blank fields,
+  repeated values, changed defaults, expired sessions, a rejected save, a
+  lost response, a duplicated save, a value the server rejects), there was
+  **no false success and no incorrect write** on either path.
+- **Coverage.** The compiler proposed an API step for 7 of the 10 workflows
+  (0 of 8 before this work). After the side-effect check below, **2 are
+  admitted** (WordPress comment, Wiki.js group). The other 5 stay on the GUI
+  for a stated reason.
+- **Speed**, median engine seconds of correct runs, interleaved timing runs:
+
+| Workflow | GUI steps | API step |
+| --- | --- | --- |
+| WordPress comment (form) | 4.15 (n=10) | 2.40 (n=10) |
+| Wiki.js group (GraphQL) | 7.16 (n=8) | 0.36 (n=8) |
+| EspoCRM account (JSON) | 4.21 (n=10) | 0.70 (n=10), not admitted: see below |
+
+## Per workflow
+
+| Workflow | Save shape | Proposed | Side effects (footprint) | Outcome |
+| --- | --- | --- | --- | --- |
+| WordPress comment | form, submit-button pair | yes | same tables, rows and fields (`wp_comments`) | **admitted** |
+| Wiki.js group | GraphQL mutation, batch of one, bearer header | yes | same (`groups`) | **admitted** |
+| EspoCRM account | JSON, three app headers | yes | the GUI also writes a `read` entry to `action_history_record` when it shows the new account; the API step doesn't | stays on the GUI |
+| Dolibarr contact | form, submit-button pair | yes | not comparable: the GUI save itself fails (see below) | stays on the GUI |
+| Medplum patient | JSON to the FHIR server's own origin, bearer header | yes | not comparable: the GUI save itself fails | stays on the GUI |
+| Redmine issue | multipart, refreshed CSRF token | yes | not comparable: the GUI save itself fails | stays on the GUI |
+| WordPress category | form fields posted by page script to `admin-ajax.php` | yes | - | stays on the GUI at run time: the page's script raises an error in the scratch run (its blocked save gets an empty answer), so it can't be checked |
+| Trac ticket | form | no | - | the page's live preview posts the form while the person types; that write can't be explained |
+| Dolibarr customer | multipart | no | - | `client` is chosen in a select widget by a click, not a recorded value |
+| NetBox IP address | multipart, form with no action | no | - | `comments` is not a recorded value, a form default or a session value |
+
+Where an API step was admitted for evaluation and ran, every run was
+correct, including the rejected, lost and duplicated saves (it halts, it is
+never retried, and a lost response ends as `RECONCILIATION_REQUIRED`).
+[results/SUMMARY.md](results/SUMMARY.md) has the counts for every workflow.
+
+Redmine's page script asks the server to refresh the form while values are
+typed (`POST .../issues/new.js`). The scratch run blocks that request, so in
+most runs the API step stayed on the GUI.
+
+## Side effects: the footprint check
+
+Repeating a save as one request skips whatever the page does around it. So
+before an API step is admitted, three GUI saves and three API saves run on
+the same inputs, the whole database is diffed around each, and the two are
+compared table by table and field by field
+(`openadapt_flow/api_footprint.py`). A column that varies between the GUI
+saves themselves (ids, times, hashes) is server-owned and not compared;
+anything else that differs keeps the save on the GUI, and `api admit
+--footprint` refuses it. On EspoCRM it found exactly the kind of gap it is
+for: the GUI path writes an audit `read` entry the API step skips. A
+workflow whose GUI save doesn't work can't be compared, so it isn't
+admitted.
+
+## Engine problems found (not API-step code)
+
+- The GUI path fails on Dolibarr contact and Trac at the save: the write
+  check reports the save "sent to a different address than the checked
+  form's" (`wire_check.py`).
+- The GUI path fails on Medplum's web app at the save: a `Page.evaluate`
+  returns a React input element ("Converting circular structure to JSON").
+- The GUI path fails on most Redmine runs (an identity check on the Create
+  button finds its target only by position).
+- Capture leaves a form's hidden CSRF token in the page HTML unredacted
+  unless that token was sent (Redmine's per-form token).
+
+## Data
+
+- `recordings/<workflow>/rec1`: the preview recordings (Oct 19).
+- `recordings/<workflow>/rec-<engine>`: the recording each workflow's
+  reported results were compiled from. Synthetic values only; the capture
+  keeps no header value, cookie or session token.
+- `results/frozen-<engine>/<workflow>/`: `compile.json`, `matrix.csv`
+  (every case, both arms), `timing.csv`, `footprint.json` (tables and
+  column names only, never values). Each run row names its engine commit.
+  Engine commits: 969cb939 (first full run), cf28c2af and f761fff8 (re-runs
+  after harness and lookup fixes), 4d977a4e (footprint check).
+- `results/preview-3cb70fa6`: the earlier preview, where 0 of 8 were proposed.
+
+The fault cases run through `proxy.py` on one origin; Medplum's save goes
+straight to its FHIR server, so it has no fault cases. Wiki.js keeps no
+server-side session, so it has no expired-session case.
+
+## Reproduce
+
+```bash
+APIGEN_FROZEN=<engine commit> APIGEN_PY=<python> ./run_app.sh <workflow> 10
+python -m benchmark.api_generalization.summarize > results/SUMMARY.md
+```

packages/flow/benchmark/api_generalization/harness.py

@@ -117,10 +117,11 @@ def start_proxy(app: AppBase) -> None:
 
 
 def cmd_record(a: argparse.Namespace, app: AppBase) -> None:
+    from playwright.sync_api import sync_playwright
+
     from openadapt_flow.backends.playwright_backend import PlaywrightBackend
     from openadapt_flow.network_capture import NetworkCapture
     from openadapt_flow.recorder import Recorder
-    from playwright.sync_api import sync_playwright
 
     out = Path(a.out)
     if out.exists():
@@ -130,7 +131,9 @@ def cmd_record(a: argparse.Namespace, app: AppBase) -> None:
     if a.tag:
         # A fresh demonstration never collides with a record an earlier run
         # made (made unique the app's own way, as each run's values are).
-        values = dict(app.unique(Case("recording", "normal", values, 1, {}), a.tag).params)
+        values = dict(
+            app.unique(Case("recording", "normal", values, 1, {}), a.tag).params
+        )
     before = app.rows()
     pw = sync_playwright().start()
     browser = pw.chromium.launch(headless=True)
@@ -234,9 +237,13 @@ def cmd_compile(a: argparse.Namespace, app: AppBase) -> None:
     if step is not None:
         # The person admitting it names the parameter that identifies the
         # record: the one the record check selects on.
-        identity = getattr(app, "api_identity", None) or {"subject_name": app.unique_param}
+        identity = getattr(app, "api_identity", None) or {
+            "subject_name": app.unique_param
+        }
         result["identity"] = identity
-        wf, admitted = admit_api_step(wf, admitted_by="benchmark-fixture", identity=identity)
+        wf, admitted = admit_api_step(
+            wf, admitted_by="benchmark-fixture", identity=identity
+        )
         result["admitted"] = admitted
         result["describe"] = describe_api_step(
             next(s for s in wf.steps if s.id == step.id)
@@ -523,14 +530,32 @@ def cmd_footprint(a: argparse.Namespace, app: AppBase) -> None:
 
     def readmit(footprint: Any) -> Any:
         wf = Workflow.load(bundle)
-        step = next((s for s in wf.steps if s.api_binding is not None and s.api_binding.kind == "web_form"), None)
+        step = next(
+            (
+                s
+                for s in wf.steps
+                if s.api_binding is not None and s.api_binding.kind == "web_form"
+            ),
+            None,
+        )
         if step is None:
             return wf, None
         clean = step.api_binding.model_copy(update={"admission": None})
-        wf = wf.model_copy(update={"steps": [s.model_copy(update={"api_binding": clean}) if s is step else s
-                                             for s in wf.steps]})
+        wf = wf.model_copy(
+            update={
+                "steps": [
+                    s.model_copy(update={"api_binding": clean}) if s is step else s
+                    for s in wf.steps
+                ]
+            }
+        )
         try:
-            wf, _ = admit_api_step(wf, admitted_by="benchmark-fixture", identity=identity, footprint=footprint)
+            wf, _ = admit_api_step(
+                wf,
+                admitted_by="benchmark-fixture",
+                identity=identity,
+                footprint=footprint,
+            )
             return wf, "admitted"
         except ApiProposalError as exc:
             return wf, f"stays on the GUI: {exc}"
@@ -549,29 +574,67 @@ def readmit(footprint: Any) -> Any:
         for k in range(a.reps):
             base = normal[0]  # the same inputs every time; only the run marker differs
             pair: dict[str, Any] = {}
-            for arm in (["gui", "api"] if k % 2 == 0 else ["api", "gui"]):
+            for arm in ["gui", "api"] if k % 2 == 0 else ["api", "gui"]:
                 marker = f"fp{k}{arm[0]}{a.tag}"
-                res = run_one(env, app, app.unique(base, marker), arm, bundle, out / f"fp{k}-{arm}", rep=k + 1)
-                pair[arm] = (res, marker, env["footprint"].get(str(out / f"fp{k}-{arm}")))
+                res = run_one(
+                    env,
+                    app,
+                    app.unique(base, marker),
+                    arm,
+                    bundle,
+                    out / f"fp{k}-{arm}",
+                    rep=k + 1,
+                )
+                pair[arm] = (
+                    res,
+                    marker,
+                    env["footprint"].get(str(out / f"fp{k}-{arm}")),
+                )
             (gres, gmark, gch), (ares, amark, ach) = pair["gui"], pair["api"]
             if not gres.get("correct") or gch is None:
-                problems.append(f"pair {k + 1}: the GUI save didn't make the expected record, so there is nothing to compare")
-            elif not ares.get("correct") or ares.get("save_actuation") != "api" or ach is None:
-                problems.append(f"pair {k + 1}: the API step didn't make the save ({ares.get('api_fallback') or ares.get('halt')})")
+                problems.append(
+                    f"pair {k + 1}: the GUI save didn't make the expected record, so there is nothing to compare"
+                )
+            elif (
+                not ares.get("correct")
+                or ares.get("save_actuation") != "api"
+                or ach is None
+            ):
+                problems.append(
+                    f"pair {k + 1}: the API step didn't make the save ({ares.get('api_fallback') or ares.get('halt')})"
+                )
             else:
-                gui.append(gch); api.append(ach); gm.append(gmark); am.append(amark)
+                gui.append(gch)
+                api.append(ach)
+                gm.append(gmark)
+                am.append(amark)
     finally:
         env["browser"].close()
         env["pw"].stop()
-    verdict = footprint_verdict(gui, api, gm, am) if len(gui) >= 2 else {
-        "equivalent": False, "differences": [], "server_owned": [], "tables": {}}
+    verdict = (
+        footprint_verdict(gui, api, gm, am)
+        if len(gui) >= 2
+        else {"equivalent": False, "differences": [], "server_owned": [], "tables": {}}
+    )
     verdict["differences"] = problems + verdict["differences"]
     verdict["equivalent"] = bool(verdict["equivalent"]) and not problems
     _, gate = readmit(verdict)
     gate = gate or "no API step"
-    report = verdict | {"gate": gate, "verdict_sha256": verdict_digest(verdict)} | frozen_check()
+    report = (
+        verdict
+        | {"gate": gate, "verdict_sha256": verdict_digest(verdict)}
+        | frozen_check()
+    )
     (out / "footprint.json").write_text(json.dumps(report, indent=1, default=str))
-    print(json.dumps({k: report[k] for k in ("equivalent", "gate", "differences", "server_owned")}, indent=1))
+    print(
+        json.dumps(
+            {
+                k: report[k]
+                for k in ("equivalent", "gate", "differences", "server_owned")
+            },
+            indent=1,
+        )
+    )
 
 
 def cmd_timing(a: argparse.Namespace, app: AppBase) -> None:
@@ -623,7 +686,16 @@ def main() -> None:
     ap = argparse.ArgumentParser(allow_abbrev=False)
     ap.add_argument(
         "cmd",
-        choices=["up", "down", "record", "compile", "footprint", "matrix", "timing", "rows"],
+        choices=[
+            "up",
+            "down",
+            "record",
+            "compile",
+            "footprint",
+            "matrix",
+            "timing",
+            "rows",
+        ],
     )
     ap.add_argument("--app", required=True)
     ap.add_argument("--out")

packages/flow/benchmark/api_generalization/recordings/README.md

@@ -0,0 +1,13 @@
+# Recorded data
+
+The demonstrations the report depends on, copied from the session scratch
+directory so a reboot can't lose them. Synthetic values only; the capture
+withholds every header, cookie and session token (`<session:...>`).
+
+- `<workflow>/rec1/`: the recording (`events.jsonl`, `network.jsonl`,
+  `meta.json`, `frames/`). The counterfactual copies (`rec1-cf`) are rebuilt
+  from these by `counterfactual.py`.
+- `<workflow>/mode.json`: the fixture proxy's fault mode when the data was taken.
+- `<workflow>/requests.jsonl`: the fixture proxy's request log (method, path,
+  status, timing; no bodies or headers).
+- `explore-*/requests.jsonl`: the proxy logs from finding each app's save request.

packages/flow/benchmark/api_generalization/recordings/dolibarr_contact/mode.json

@@ -0,0 +1 @@
+{"mode": "pass", "path_regex": "^/contact/card\\.php$", "body_exclude": "", "body_require": ""}