Repository navigation
test: safety net for decomposing the largest files (PR 1) - #1510
Conversation
The safety net for the decomposition plan (DECOMPOSITION.md, PR 1): - scripts/check_file_size_ratchet.py keeps the ten largest files from growing past the budget table in the plan; - scripts/verify_moved_code.py proves a move commit only relocates code (identical ASTs, new code limited to imports, re-exports and protocol stubs) and lists test patches a move would make stale; - tests/test_import_surface.py pins the names replayer, __main__, qualification and ir provide, and Replayer's methods. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each flow-unit shard keeps its coverage data; flow-coverage combines them and writes the safety-path number to the job summary. It is not in packages-gate: the floor is set at the measured value once known. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Nine scenarios replay one committed MockMed triage bundle under different drifts through the real Replayer and PlaywrightBackend: clean passes (with and without the structural rung), identity halts (missing, look-alike, growth, sort), a modal postcondition halt, the irreversible-step risk gate and theme healing. Each compares the normalized report, the hash of every file the run wrote, the backend action trace and the live app state with a baseline. Writing a baseline runs the scenario twice and refuses values that differ between identical runs. Baselines are Linux results generated in the flow-critical job; unit shards skip the scenarios. CODEOWNERS covers the packages the plan adds. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reverted once the baselines from this run are committed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Baselines and the recorded MockMed triage bundle come from the flow-critical job of CI run 40123224669 (write mode, each scenario run twice and identical). Ruff skips the bundle's generated workflow.py, which is part of the bundle digest. Ratchet budgets reset to origin/main ecc162e. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The baselines are committed; golden runs now compare. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…larations A protocol attribute such as Optional[dict[str, Any]] was read as an assignment because its dump contains 'value='. Statements now record whether they only declare (an annotation with no value, or a docstring) when they are parsed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ode reads A moved method reads module-level names from its new module, so a test that patches old_module.name (an import such as resolve, or a constant) no longer reaches it. The verifier now tracks the names each moved function reads and flags a test that patches one through the old module without also patching it where the moved code reads it. Found by the first large replayer move, where a scroll-readiness test patched runtime.replayer.resolve. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in packages.yml: keep main's xdist shards, fast lane and single critical e2e run; re-add the golden-runs step, the per-shard coverage data and the non-gating flow-coverage job on top.
Budgets are each file's line count on main at 213f3d9. The snapshot adds the names and Replayer methods main gained (276 methods); none were removed.
They were written against main of October 24. CI regenerates them, bundle included, from the merged tree in the next commits.
Reverted once the baselines from this run are committed.
Written by Packages run 41704998213 (workflow_dispatch, flow core critical, OPENADAPT_GOLDEN_WRITE=1, Playwright 1.61.0). In write mode each scenario ran twice and the two results were equal before the baseline was written. Copied unchanged from the flow-golden-runs artifact.
This reverts commit 3881357.
…selines flow-coverage now installs coverage 7.15.1 (the version packages/flow/uv.lock pins) from .github/ci-tools/uv.lock with hashes, as tests/test_ci_locked_installs.py requires. public-artifacts.json lists the committed golden baselines (regenerated with --write-public-artifact-inventory).
With --cov in every flow-unit shard, the full lane's shard 2 hit its 40-minute limit at 99% (Packages run 41715793206): coverage slows the suite by about a third. Pull requests and main's post-merge run now collect none; a schedule or a dispatch collects it, with a 60-minute limit, and flow-coverage reports it.
Conflict in packages/flow/pyproject.toml: take main's comment about the benchmark harness and keep tests/golden_runs/baselines in extend-exclude.
replayer.py grew to 19524, __main__.py to 12915, ir.py to 6352, recorder.py to 4823 and playwright_backend.py to 5199 on main while this waited. The snapshot adds the 4 names main added; none were removed.
… merge Three files grew on main since the budgets were set (replayer.py +5, __main__.py +320, playwright_backend.py +83); section 6 sets each budget to main's size when this PR merges. The import-surface snapshot drops _refuse_rollback_durable, which #1772 removed on main with no remaining references, and adds the names #1772 added.
Independent adversarial review: BLOCKReviewed [P2] Reject executable code inside allowed statements. verify_moved_code.py:77–95 ignores an entire [P2] Preserve the number of identical code units. verify_moved_code.py:240–244 checks AST membership without consuming occurrences. With identical methods in two classes, deleting one class and its method still returns success: the surviving method satisfies both old units. Compare occurrence counts so a surviving copy cannot hide a deletion. Validation: 67 root/workflow tests and 19 Flow tests passed, including all nine real headless Linux golden scenarios. The snapshot-regeneration test was skipped with write mode disabled. Three independent regression probes failed on the false passes above; unchanged-code and direct-body-edit controls passed. Ruff and formatting passed on all eight new Python files. No reviewed source or baseline was changed. |
Ratchet budgets raised to main's current sizes and the import-surface snapshot regenerated (FIELD_REGION_SIZE was removed on main by #1831).
Independent adversarial review: BLOCKBLOCK remains at fbc0e19. The main merge leaves the previous verifier blockers unchanged: both The verifier still skips the entire I reconciled current source with the earlier measured review, without rerunning unchanged probes or golden scenarios. Metadata-only Git counts show all ten ratchet budgets equal the files' sizes at this head; that is not a checker or CI execution claim. The new workflow/import-pin merge does not fix the verifier defects. No per-app engine change is involved; the defect class is incomplete code-preservation evidence. Held-out runtime evidence is not established by this review. Review by Codex. |
The move checker accepted three kinds of hidden new code and one hidden deletion: - An `if TYPE_CHECKING:` block was skipped whole, including its `else` branch, which runs at import time. Both branches are now walked statement by statement; only imports inside them are ignored. - `name = make().value` counted as a re-export because its outer value is an attribute. A re-export must now be a plain dotted name. - Old units were matched by membership, so one surviving identical copy satisfied two old ones. Each old unit now consumes its own copy: first in its own file, then from copies left over in changed or new files. - Same class of gap: a stub with a call in a decorator, default or return annotation, a declaration whose annotation calls, and a class base or metaclass built by a call ran code while passing as non-logic. These are now new code. The pure-move test now counts 7 stubs: the two branches of the mixin's TYPE_CHECKING base alias are reported as re-exports instead of skipped. Ratchet budgets follow main's current sizes: replayer.py 19852 -> 19859 (#check-to-click settled recheck on main), ir.py 6505 -> 6514 (TokenTemplate.bearing and CheckToClickRecheck.settled_on on main), interactive_recorder.py 4807 -> 4603 (shrunk on main; budget lowered so the gain is kept). No snapshot was regenerated: the import-surface snapshot regenerated from this head is byte-identical, and the golden baselines are Linux-only and are checked by CI.
…main All nine golden runs failed on the merged head in CI (run 42557457547), which caught a real serialization change from main: TokenTemplate gained an optional `bearing` field (identity-bearing tokens). The new baselines are that CI run's own outputs (OPENADAPT_GOLDEN_OUT), produced on Linux with Playwright 1.61.0 from this branch's head. The head adds no product code, so this is main's behaviour. What changed, in every scenario, and nothing else: - report.workflow_contract_sha256 and outcome_envelope .workflow_contract_sha256: the committed bundle's workflow dump now carries `bearing: null` on each identity token. Recomputing the hash with `bearing` dropped gives the old value 4e2641e2... exactly; with it, the new value 03ef6827.... - postcondition_evidence[*].contract_sha256 and step_contract_sha256 (134 entries): derived from the same workflow dump, so they move with it. - mockmed_rename and mockmed_theme only: report.results[5].heal old_anchor/new_anchor identity_template tokens gain `bearing: null` (12 + 12 tokens), and heals/step_005/heal.json's hash changes because it serializes the same anchors. No outcome changed: success, halted step, app_state (hash, banner, encounter item), the backend action trace, the environment, and every other artifact hash are identical to the old baselines.
tests/test_flow_claims_ci.py (from main) reads flow-unit's last step as the junit claims upload. This branch's nightly coverage upload came after it, so the launcher test failed. The coverage upload now runs just before it; both still run under always().
|
From: Codex BLOCK — conditional execution scope is still lost in Both branches of a Independent tests used real temporary Git repositories, the checker API, its CLI, and execution of the before/after Python sources. All six moved cases returned The three earlier findings were reconstructed from the public review comment because their original executable probes were unavailable. Head and literal parent refuse all three; supported pre-fix 98eb0e4 accepts them incorrectly. I do not claim byte identity with the old probes. Validation: head84 cases =77 passed,6 failed,1 original skip; parent84 =76 passed,7 failed,1 original skip; pre-fix5 =2 passed,3 failed. The parent's additional failure is the original workflow artifact-upload condition assertion; head passes it. Total173 cases/519 setup-call-teardown reports. Scoped root/Flow Ruff, formatting and Python3.10 syntax checks pass. Actual Python3.10 execution and full golden-scenario execution were not run. The nine changed golden baselines match the corresponding CI-produced bytes; direct rehashing of their referenced heal artifacts remains unmeasured. Generality: no per-app engine code is added. The blocker concerns conditional execution provenance across three sibling unit kinds. The independent synthetic cases extend the motivating examples; no held-out real-app claim is made. |
…identity Both branches of an `if TYPE_CHECKING:` block were walked with the same prefix, so an unchanged statement, function or method that moved from the `if` branch to the `else` branch (or back) still matched and the check passed, although the code now runs (or exists) in the other case. units() now carries the chain of enclosing TYPE_CHECKING branches (`[TYPE_CHECKING:if]`, `[TYPE_CHECKING:else]`, nested in order) into every unit's comparable dump: statements, functions, methods, class decorators and call-built class bases alike. A unit that changes branch is a deletion plus new code. Moves between files that keep the branch still pass. Stub and re-export recognition moved into units() as the unit's is_declaration flag, because the old file-wide sets of bare dumps no longer match branch-scoped dumps. Tests: Codex's six moved cases (statement, function, method; each direction) fail, and the six unchanged-branch controls pass. All six moved cases pass on the previous script. Merged with main also needed: - __main__.py ratchet budget 13953 -> 14119 (main added the local-path and sealed-set inventory helpers). - import_surface.json regenerated from this head: it adds _PLAINTEXT_ARTIFACT_MARKERS, _complete_inventory, _inside_sealed_set and _local_path_kind, which main defined at the top level of __main__.py. Nothing was removed.
|
From: Codex APPROVE for the ordered conditional-execution scope fix. The earlier branch-scope blocker is resolved. All65 checker/CLI probes pass on this head: the original17 plus48 covering statements, decorators, call-built bases/metaclasses, ordered nesting and same-chain cross-file moves. The identical tests on the literal parent produce43 passing controls and22 expected semantic assertion failures. For those22, actual execution counters change while the parent checker and CLI accept; this head refuses the moves. This comparison uses supported APIs and temporary Git repositories. The complete candidate batch has143 passes and one original regeneration skip across144 cases/432 phases. The first parent author batch has24 passes and one unchanged size-ratchet failure (14119 lines against13953); it correctly stopped. The separate parent-only probe comparison preserves that failure and does not claim a fully green parent. Across both runs:234 cases/702 phases,210 passes,22 expected parent probe failures, one retained parent ratchet failure and one original skip. Item/JUnit/phase records,177 source/ref hashes and named dependency parity were reconciled independently. Root/Flow Ruff, formatting and Python3.10 syntax rules passed; actual3.10 execution and Mypy were not run. All33 golden-baseline tree entries, including the nine previously reviewed baseline objects, are unchanged from the previous review. No fresh golden replay or direct rehash of underlying heal artifacts is claimed. Required public CI remains a merge gate. No per-app engine code: yes. Fix the class: one ordered execution-context model covers the supported move entry points and positive controls. Held-out real-app evidence: not established; this is a scoped source-preservation review. |
The CLI census requires allow_abbrev=False on every parser.
|
Retriggering CI: the last run tested a merge ref from before #2122 fixed main's gitleaks. |
Main grew these files after the budgets were set; this pull request adds no product code (git diff origin/main -- packages/flow/openadapt_flow is empty), so the budgets move to main's sizes at bfdba71: - runtime/replayer.py: 19859 -> 19861 - __main__.py: 14119 -> 14174 - ir.py: 6514 -> 6551 - interactive_recorder.py: 4603 -> 4606 - backends/playwright_backend.py: 5986 -> 5990
… main Main's e604531 (fix(identity): separate strict verification from OCR noise scores, #1961) changed these two outcomes. Both drifts heal the 'Open' step (step_005), whose committed bundle carries a legacy identity template. Under #1961, verify_template_identity on that template and the refreshed live band ('Jane Sample Knee pain referral High') returns abstain with coverage 0.0, where e604531^ returned verified, so the heal governor quarantines the heal and the run halts. The new outcome halts before any effect: - execution_outcome HALTED, transaction_outcome HALTED_BEFORE_EFFECT; - effect_journal and change_ledger empty; - app_state: no banner, no encounter item, hash '#patient/p1'; - the last delivered action is the reversible 'Open' click, made after structured identity verified the right patient. 'Save Encounter' is never reached. Before, rename halted at the irreversible Save step and theme saved the encounter (COMPLETED_UNVERIFIED). The new files are the CI outputs (OPENADAPT_GOLDEN_OUT) of run 42669375410 attempt 4, on Linux with Playwright 1.61.0, from this branch merged with main at bfdba71. They are byte-identical to the outputs of run 42664453883, which ran on an earlier main, so two runs agree. The other seven scenarios' outputs from the same run match their committed baselines byte for byte.
|
Codex delta review: APPROVE at 78c76ad. The engine tree matches main. The delta is five line budgets and the two regenerated golden JSONs. Both record HALTED_BEFORE_EFFECT, with an empty effect journal and change ledger, and Save Encounter is never reached. |
…unts main grew __main__.py (+313), ir.py (+27), interactive_recorder.py (+14) and playwright_backend.py (+6) after this table was written; this branch does not touch those files.
|
Codex delta review: APPROVE at 41e31d2 for the four budgets, which equal main's actual line counts. The main merge reproduces a clean three-way merge. |
Summary
This is PR 1 of the decomposition plan (
packages/flow/docs/design/DECOMPOSITION.md, section 4.3). It adds the safety net that every later move is checked against. It changes no production code.scripts/check_file_size_ratchet.pyruns in Python CI throughtests/test_file_size_ratchet.py. It fails when a listed file grows past its budget in the plan's table. A PR that adds net lines raises the budget in its own diff, with a one-line reason. The table and section 6 came from docs(flow): decomposition plan for the largest files #1497. This PR adds the script and the test; there is only one ratchet.scripts/verify_moved_code.pychecks that a move commit only relocates code. Every function, method and statement in the base must exist with an identical AST in the head. New code may only be imports, re-exports and protocol stubs. It also lists test patches that a move would make stale.packages/flow/tests/test_import_surface.pypins the names thatreplayer,__main__,qualificationandirprovide. It also pins the 276 methods ofReplayer.packages/flow/tests/golden_runs/runs nine scenarios with the realReplayerandPlaywrightBackendagainst MockMed. Each run's normalized report, file hashes, backend action trace and live app state must match the baseline.flow-coveragejob combines the unit shards' coverage data and reports safety-path coverage in the job summary.This round
packages.yml. Main's xdist shards, fast lane and single critical e2e run are kept. The golden-runs step and the coverage job are added on top.workflow_dispatchPackages run (41704998213) then recompiled the MockMed bundle and wrote all nine baselines on Linux with Playwright 1.61.0. Write mode runs each scenario twice and refuses results that differ; all nine matched. The baselines were copied unchanged from theflow-golden-runsartifact, and the temporary commit was reverted. Two later runs in compare mode reproduced all nine: 41715793206 and 41733145873, the second at the current head. No baseline was edited by hand.replayer.pyis at 19,303 lines. Main keeps changing these files, so re-runpython scripts/check_file_size_ratchet.pyjust before merging and update any number it reports.Replayermethods, up from 260. No name was removed.flow-coverageinstalls coverage 7.15.1 from a newcoveragegroup in.github/ci-tools/uv.lock, with hashes. That is the versionpackages/flow/uv.lockpins. The earlier unhashedpip installfailedtests/test_ci_locked_installs.py.packages/flow/public-artifacts.jsonnow lists the baseline files.Merge with main, round 2 (head 5daa5b9)
packages.yml, in the flow-unit job. Main now picks shard files withscripts/ci/flow_shards.pyand runs-n logical. The merge keeps both, plus this PR's golden-runs exclusion from the shard list, theCOVERAGE_*env, the upload step and theflow-coveragejob. actionlint passes.replayer.pyby 5 lines (to 19,529),__main__.pyby 320 (to 13,235) andplaywright_backend.pyby 83 (to 5,282). This PR leaves all three unchanged from main. The other seven budgets already matched.OPENADAPT_WRITE_IMPORT_SURFACE=1._refuse_rollback_durableis dropped from the__main__pins because Rollback on durable runs, and after a crash (#1699) #1772 removed it on main and nothing references it any more. The five names Rollback on durable runs, and after a crash (#1699) #1772 added are now pinned.src/screens/Stops.test.tsxfailed in the Desktop frontend job ("answers the current stop with one key"). This PR doesn't touchpackages/desktop. The same timing flake is what desktop: prepare review card input before paint #1825 fixes.Tested at this head:
python scripts/check_file_size_ratchet.pypasses; rootpytest testsgives 805 passed, 14 skipped; flowtests/test_import_surface.pygives 6 passed andtests/golden_runsgives 5 passed, 9 skipped (the Linux-only scenarios); ruff passes on the golden-runs tests.Notes for review
generate/verify_contract.py(4,157 lines) is now larger thanterminal_verification_v2.py(3,663), which is in the table. The table keeps the ten files from docs(flow): decomposition plan for the largest files #1497. Swapping one in is a separate choice.src/screens/NeedsYou.test.tsxfailed in the Desktop frontend job. This PR does not touchpackages/desktop, so the failure is outside it.Test plan
pytest tests/test_file_size_ratchet.py tests/test_verify_moved_code.py(13 passed)test_ci_locked_installs,test_ci_required_check,test_nightly,test_packages_concurrency,test_newest_green(52 passed)