diff --git a/FINAL_REPORT.md b/FINAL_REPORT.md index 74cdc96d..f815161b 100644 --- a/FINAL_REPORT.md +++ b/FINAL_REPORT.md @@ -1,111 +1,412 @@ -# Final report: microcosm #462 register alignment +# Final report: pkg3 r4 — main merge resolved by porting the calibration policy into the F0 spec + +2026-08-23. This section reports the r4 continuation (the `origin/main` +merges and the F0 home for the post-transfer calibration policy). The +receipt-validation report it supersedes as the branch's latest result +remains below as history. ## Outcome -Completed the split-PR remediation on `loss-contract-alignment`, based on -`origin/main` at `7b6e10b`. The change is now register alignment only: one -shared critical-target register, one shared congressional-district classifier, -two consumers, builder contract-row gating, and behavioral containment of the -publish contract. - -The critical-row loss multiplier was removed entirely per -[microcosm#492](https://github.com/PolicyEngine/microcosm/issues/492). There is no -constant, CLI option, validation, loss overlay, telemetry, diagnostics/scorer -provenance, or historical replay pin left. `_fiscal_target_loss_weights` is -source-identical to `origin/main`, and its output therefore preserves main's -bit-level behavior for the same registry and family multipliers. - -## Sol round-1 findings - -1. **Table 1.4 selector parity:** removed the builder-only - `accepted_name_prefixes=("irs_soi.",)` constraint. The adapter now has - exactly the shared requirement's substring and suffix selectors. The - outside-prefix reproduction is builder-rejected. -2. **Congressional-district parity:** added exported, stdlib-only - `is_congressional_district_target(name, metadata)` and made the publisher - and builder classifiers thin wrappers. It ORs layout dimension, source-id - token, geography level, geography scope, truthy CD GEOID, and name token. - The builder's exact/semantic, Table 1.4, and zero-support paths now see the - same registry metadata. -3. **Recorded relative-error shape:** a matched row with missing/`None` - `relative_error` now fails with the publish-contract message instead of - silently passing after recomputation. Existing non-numeric and stale-value - checks remain. -4. **Behavioral anti-drift:** the load-bearing test now runs adversarial rows - through both consumers for exact-name, family+role, Table pattern, - missing/non-finite values, and a disallowed incumbent escape at the 0.25 - hard stop. A production Ledger compile supplies six separate CD evidence - rows; builder and publisher exclude identical six-name sets and counts. - Field comparisons remain as fast checks, and any added conjunctive prefix - is proven to trip the guard. - -The [#490](https://github.com/PolicyEngine/microcosm/issues/490) medical 0.25 -adjudication tolerance and its adjacent comment in `us_critical_targets.py` -remain byte-for-byte unchanged, as required. - -## Reproduction receipts - -The Table 1.4 prefix reproduction now returns: +PR #742's source conflict against the locally available `origin/main` is +resolved by three merge commits: `e66074ad` (main at `b4dfa0e7`, carrying the +F0 policy port), `e0947020` (main at `2aa96795`, #733), and `1fc9055e` (main at +`055dcfaf`, #740). `origin/main` at `055dcfaf` is an ancestor of the final +tree. Commit `87e638bd` closes the three deterministic merge-union guards +exposed by the complete build shard. The full five-shard inventory, +repository-wide Ruff, coverage attestation, and final wheel boundary are +green. + +The branch's post-transfer calibration policy declaration — formerly in +`specs/us_imputation_lineage.yaml`, which main's F0 migration deleted — +now lives where imputation models are authored: + +- a closed, typed exact variant of `regime_gated_qrf_model` carrying the + `post_transfer_calibration_policy_v1` payload + (`packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json:746-807`); +- the authored declaration in + `packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml:94-192`, + with its generator in `tools/us_bundle_generation/imputation.py:269,2337`; +- a ninth `post_transfer_calibration` stacked-authority component projected + at authority binding version 11 + (`spec_engine/stacked_authority_semantics.py:414,466,618-622`; + `us/spec/battery.yaml:824-826`); +- the spec-matches-code identity test re-ported to the generated-bundle + boundary in + `packages/microcosm-build/tests/test_imputation_lineage_spec.py:99-103`, + combined with main's producer-registry test, holding the authored payload + byte-equal to `post_transfer_calibration_policy_identity` + (`us_runtime/post_transfer_calibration.py:301`) in code. + +No comparator, band, threshold, seed, fold, gate, or sample contract +changed. No host pool build ran. Nothing was published and PR #742 was not +merged. This report-closing commit is the payload for the single permitted +push to `origin/battery-pkg3-two-part`. + +## Anti-rot chain + +The port adds 92 authored configuration fields. Every count, sha, and pin +site the chain covers was walked and is test-green on the final tree: + +- `EXPECTED_CONFIGURATION_FIELD_COUNT = 41_471` + (= 32,252 authored + 9,219 resolved; + `spec_engine/field_usage.py:29-31`), with the mode/effect counts at their + matching literals and the `imputation_models` usage claim moving from 4 + fields to 96 with claim SHA-256 + `e4d6b6b747fcec1c027e0f1c2d1905274c0426217a61383b02e75baadb93db4d` + (`field_usage.py:359-362`). +- The pointer-inventory sha and the regenerated + `docs/evidence/spec-engine/us-f0-coverage.json`: + `tools/spec_engine_coverage.py --check` reports 41,471/41,471 + configuration fields and 40/40 inventory checks with no drift. The pinned + inventory SHA-256 is + `2daa3ee07ac2e5d5ab731348edbca8c7a58438e9819d1ac3707070c7500a1c63` + (`tools/spec_engine_coverage.py:42-45`). +- Test literals in `test_spec_engine_field_usage.py` and + `test_spec_engine_coverage_tool.py`. +- Bundle identity pins. The US pin + `d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0` in + `test_us_multispine_pool_tool.py` was recomputed via + `load_bundle("us").spec_sha256`. The loader golden vector moved to + `b1ab6ab000689cc03e1088d422d7b6328b9bff2878f39448a529e5513c03ed14` + (`test_spec_engine_loader.py:237-239`); single-edit bisection proved the + mover is the branch's edits to seed-attested kernel modules (the seed + protocol digests attested kernel bytes into the resolved bindings inside + the hashed envelope, `spec_engine/loader.py:404-418`), not the shared + schema edit — the legitimate identity movement `CLAUDE.md` documents. + +## Merge resolutions + +Merge 1 (`e66074ad`, main at `b4dfa0e7`): `specs/us_imputation_lineage.yaml` +stays deleted (lineage derives from the authored US bundle); +`tools/build_us_multispine_pool.py` keeps main's immutable-safe direct +dataclass walk plus this branch's omission of empty `target_regimes` in both +the mapping and dataclass paths (`tools/build_us_multispine_pool.py:3826-3847`); +root journals are unioned and historicized; the port and chain above are +folded in. `test_imputation_lineage_spec.py` combines main's generated +producer-registry assertion with the branch's policy-identity assertion. + +Merge 2 (`e0947020`, main at `2aa96795`): sole conflict was the UK entry in +`test_spec_engine_country_bundles.py`. Both parents' pins were computed on +trees missing the other side's envelope movers (this branch's +stacked-authority version-11 binding vs #733's UK sources/runtime +retarget), so the union pin was recomputed fresh via `load_bundle("uk")`: +`bb7110699a9cb7a346ecd478f55b7a1c57bdc0c3f3283705b8a9b51830207193`. BE +(`bf022118…`) and US (`d3de6760…`) recomputed identically on the union — +the `b4dfa0e7..2aa96795` range touches no BE- or US-side envelope input, +and its `sources.schema.json` edit sits in the grammar receipt outside the +hashed envelope (`spec_engine/loader.py:227-229,401`). Main's re-cut UK +gate-battery digests and `microcosm-data` contract pins merged clean and +pass unchanged. + +Merge 3 (`1fc9055e`, main at `055dcfaf`): the sole conflict was again the UK +country-bundle identity after #740 added the E8 UK spec stages. Fresh +union-tree `load_bundle` calls produced BE `bf022118…`, UK `8bf62b6e…`, and +US `d3de6760…`; the UK test pin was resolved with the exact union value +`8bf62b6e47583da1bdad1b71be1e705f424e6e245880e90f4411aba57fa5eb93`. +Main's UK gate/data pins merged cleanly. + +The complete build shard then exposed three exact union misses, fixed in +`87e638bd`: the direct dataclass serializer now applies the empty +`target_regimes` omission (`build_us_multispine_pool.py:3841-3847`); the +authored-SHA audit allows only the exact policy-identity SHA path in addition +to the two external-asset pins (`test_us_spec_bundle.py:707-770`); and the +closed runtime import graph is pinned at 66 after the branch-added +`post_transfer_calibration.py` module (`test_us_spine_blindness.py:3270-3315`). + +## Verification on the final tree + +- The final focused policy/count/pin/gate/data/serializer/SHA/import-graph + suite passes, as do the two unchanged trade-publication crash tests after + shared-host load fell below the point where their child CLI import exceeded + the existing 60-second bound. No timeout was changed. +- Full five-shard suite on the final union. Fit, calibrate, frame, and data ran + in one pytest process each. The build root's single-process behavioral run + was green (6,248 passed, 39 skipped) but reached 15.989 GiB, so that resource + receipt was rejected. The authoritative capped rerun covered all 262 build + test files in 17 fresh pytest processes, reproduced the exact 6,248/39 + aggregate with exit 0, and peaked at 10.284 GiB; no guard split or + intervention was needed. + + | Shard | Result | Peak RSS | + | --- | --- | ---: | + | `microcosm-fit` | 93 passed, exit 0 | 0.862 GiB | + | `microcosm-calibrate` | 203 passed, exit 0 | 0.449 GiB | + | `microcosm-frame` | 294 passed, 36 skipped, exit 0 | 6.492 GiB | + | `microcosm-data` | 275 passed, 1 skipped, exit 0 | 11.052 GiB | + | `microcosm-build` | 6,248 passed, 39 skipped, exit 0 | 10.284 GiB | + + Build authoritative receipt: `files=262 batches=17 passed=6248 skipped=39 + exit=0 max_rss_bytes=11042193408 guard_gib=12.0`. +- Repository-wide `uv run ruff check .` and `git diff --check`: pass. +- Final wheel boundary: all five shard wheels build offline. Reinstalled into + the existing clean, lock-constrained wheel venv, all five namespaces import + from its site-packages prefix with `policyengine_us` absent; + `tools/spec_envelope_digests.py be uk` runs there; and installed-wheel + `load_bundle` reproduces exact BE `bf022118…`, UK `8bf62b6e…`, and US + `d3de6760…` identities. A second brand-new offline venv could not resolve + uncached third-party packages, so no network-dependent claim is made. + +## Boundaries + +This lane ran no pool build (the host queue owns those builds; the 1% baseline +at `_buildo-runtime/out/battery-verify/baseline1pct/` stands for +before/after diffs). Certification, publication, and release-chain +mutation remain with their existing owners. PR #742 is not merged. + +--- + +# Final report (historical): pkg3 post-transfer receipt validation failure #2 + +## Outcome + +Fixed at executable commit `a932974f`. + +The supplied weeks-unemployed failure was a receipt-generation bug, not an +invalid carrier model and not a count-target exception. Candidate capacity and +prefix selection reduced the same ordered weights through different float64 +paths, so the prefix exceeded its declared candidate mass by one ULP. The +required cross-target audit then exposed the same class of bug one level up: +the first repair composed maximum capacity from independently rounded +partition endpoints, putting both child-support maxima 42 ULP above the whole +recipient mass. + +The complete fix changes both generating relationships: + +- one immutable `_PrefixSchedule` now supplies each candidate endpoint and all + prefix-selection evidence (`post_transfer_calibration.py:282-287,471-515,844-872,891-928`); +- maximum capacity is generated from the row union + `fixed_positive | allowed_positive | zero_candidates`, zero-masked onto the + recipient-weight vector with identical length, order, and reduction topology + to `recipient_total` (`post_transfer_calibration.py:823-885`). + +The exact validator inequalities are unchanged: maximum must not exceed its +recipient superset, and a reported prefix must not exceed its candidate set +(`post_transfer_calibration.py:1457-1475,1499-1529`). No tolerance, threshold, +band, gate, comparator, seed, fold, target, or carrier constraint changed. + +Both SHA-pinned no-build harnesses now validate. All five package test roots, +repository-wide Ruff, touched-file formatting, and whitespace checks are +green. This lane ran no host build and made no push. The serial host owner owns +the next 1% rerun against `a932974f`. + +## No-build checkpoint reproduction + +The read-only stage root was: ```text -SOI Table 1.4 national dollar fit failed: other.table_1_4.all.bad_amount@2024: relative_error=1 exceeds 0.25 for SOI Pub 1304 Table 1.4 national dollar rows (soi_table_1_4_national_dollar_rows); target=100.0, final_estimate=200.0. +/Users/maxghenis/PolicyEngine/_buildo-runtime/out/battery-verify/pkg3/pool.checkpoints/stacked/8f5077d6a1d5440b241f22fe4d20ad1d889924a27d094cb669e1035f9306546b ``` -The missing-relative-error reproduction now returns: +The current no-build checks are: ```text -SOI Table 1.4 national dollar fit failed: irs_soi.ty2023.table_1_4.all.adversarial_amount@2024: missing recorded relative_error; the publish contract requires a numeric value. +uv run python tools/reproduce_us_post_transfer_weeks_checkpoint.py \ + --checkpoint-stage-root --expect valid +uv run python tools/audit_us_post_transfer_child_support_checkpoints.py \ + --checkpoint-stage-root --expect valid ``` -The CD reproduction has the owner-mandated exclusion result: +### Weeks-unemployed failure + +`tools/reproduce_us_post_transfer_weeks_checkpoint.py` validates the assembled +Frame plus unemployment-compensation and weeks target file, identity, and raw- +draw hashes. It reconstructs only the native clone-0 vectors and calls the live +kernel and strict validator; it performs no fit, DAG execution, artifact write, +or build (`reproduce_us_post_transfer_weeks_checkpoint.py:1-6,27-58,76-143,146-254`). + +Pinned artifacts: + +| Artifact | File SHA-256 | Identity SHA-256 | Raw-draw SHA-256 | +| --- | --- | --- | --- | +| `assembled.checkpoint.h5` | `5ce1815fc44dc43c7c24ccf27526852b8f1bddbdfe371255410a22f9b56ac015` | whole file pinned | n/a | +| `019__unemployment_compensation.h5` | `dc6637936ed4bd0322d38eaa3a4920fd137565f314387db3b3fdc7dfd6bc3086` | `708722093ca610426175998d50bbb6663585b07ffef912899f17adc90520f51f` | `e32d1559668e10b24abad8e1d639e4dbade964a712925bfe8f56d3136b839840` | +| `000__weeks_unemployed.h5` | `898397733aa3e5d8ec7d6679cb16a0504e826e25d23ca2c788f4397e0e061a43` | `d0d554ba05045e39a07f0f9515c83bbf754f067df12b8247f4bf3866162c4bdd` | `0214c8dcbc118676336069b906a07ee6145f2178542b6c5b4fb5899ad62d09f3` | + +The production owner selects ASEC clone-0 reference rows, ACS clone-0 +recipient rows, transferred nonnull mutable cells, and positive-UC mutable rows +for both the weeks allowed and addition masks +(`stacked_spine.py:8960-8971,8995-9032`). The pinned replay contains: + +- 38,604 native person rows; +- 4,311 reference and 34,293 recipient/mutable rows; +- 134 positive reference rows; +- 24 initial recipient positives, all disallowed; and +- 32 positive-UC addition candidates + (`reproduce_us_post_transfer_weeks_checkpoint.py:146-193`). + +At reproduction commit `4cc41652`, the harness with `--expect invalid` exits +zero only for the exact supplied failure. Candidate capacity is +`85,676.23791782455`; the ID-ordered upper prefix is +`85,676.23791782456`; the excess is `1.4551915228366852e-11`. The sole false +relationship is: + +```text +upper_prefix_mass <= addition_candidate_mass +``` + +It raises exactly: ```text -builder_excluded=True -publisher_excluded=True -builder_failures=[] +ValueError: Frame post-transfer calibration person/source_operator_weeks_unemployed/weeks_unemployed: match-reference carrier capacity relationships are invalid. ``` -Calling that row "rejected" would contradict the required OR-union exclusion -semantics. The two malformed critical rows are rejected; the CD row is -symmetrically excluded by both consumers. +The harness pins that predicate, both floats, and the complete error rather +than accepting any aggregate validation failure +(`reproduce_us_post_transfer_weeks_checkpoint.py:210-231,257-286`). + +Against `a932974f`, `--expect valid` reports candidate and upper prefix both +`85,676.23791782456`, zero prefix/candidate delta, no failed relationships, +and a valid receipt. The attainable-union maximum under the recipient +reduction topology is `85,676.23791782453`. + +### Child-support cross-target reproduction + +`tools/audit_us_post_transfer_child_support_checkpoints.py` reconstructs the +same native clone-0 support and half weights for both child-support targets. +It pins each target's whole-file, identity, and raw-draw hashes, then requires +both receipts—not merely one—to match the requested exact red or green state +(`audit_us_post_transfer_child_support_checkpoints.py:1-80,91-207,210-302`). + +| Artifact | File SHA-256 | Identity SHA-256 | Raw-draw SHA-256 | +| --- | --- | --- | --- | +| `000__child_support_expense.h5` | `d119075e19fb767f3d8d24c7c0149d0df1ed963774a4b93d96974a72b3ac9bfe` | `41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab` | `8b2845aff0aa0695d98ae30828523bf6bca9c5d4ed5d2d91d2d1a636bb917600` | +| `001__child_support_received.h5` | `66120896d5793f3d737f9ffac2058e2196992e357f8d869f4b31b259d041b3aa` | `41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab` | `ea7f2eebb430b654acc639ef6ee6ed482207ffd74d54ba3a47cb55056813a381` | + +Against the incomplete candidate-only repair `d7b12bab`, both receipts fail +only: + +```text +maximum_attainable_mass <= recipient_total +``` + +For each target, recipient total is `79,926,522.10879111`; the partition- +composed maximum is `79,926,522.10879174`; the excess is +`6.258487701416016e-07`, or 42 ULP. The underlying endpoints are: + +| Target | Allowed-positive mass | Addition-candidate mass | Old composed maximum | +| --- | ---: | ---: | ---: | +| `child_support_expense` | `71,696.09739141785` | `79,854,826.01140033` | `79,926,522.10879174` | +| `child_support_received` | `180,209.75664861224` | `79,746,312.35214312` | `79,926,522.10879174` | + +Against `a932974f`, both strict receipts validate. Their attainable-union +maximum equals recipient total exactly, `79,926,522.10879111`, while the +independently rounded diagnostic partition sum remains +`79,926,522.10879174`. This demonstrates that the generating set relationship, +not the validator, was repaired. + +## Root cause and semantic decision + +Float64 addition is order-sensitive. The original code used a masked +`ndarray.sum` for candidate capacity and a separately ordered `np.cumsum` for +prefix selection. The first repair correctly unified those two values, but it +then added independently rounded fixed, existing-positive, and zero-candidate +endpoints to describe a different claim: the maximum mass of their row union. + +Both strict invariants are semantically correct: + +1. a prefix cannot exceed the candidate set from which it was selected; and +2. an attainable subset cannot exceed its recipient superset. + +The final maximum implementation retains the recipient vector's length and +order and replaces unattainable entries with zero. With finite nonnegative +weights, each attainable leaf is less than or equal to its corresponding +recipient leaf, and the identical reduction topology preserves that ordering +through every floating-point addition (`post_transfer_calibration.py:775-783,823-885`). +It does not use `min`, `nextafter`, a tolerance, or a post-hoc clamp. + +The regressions lock both numerical mechanisms for every late +`match_reference` declaration: + +- the exact 32 production weeks weights distinguish masked sum from ordered + prefix by one ULP; +- a four-weight case makes independently rounded capacity partitions exceed + their whole set by one ULP; and +- a constrained proper-subset case makes a compressed subset sum + `0x1.433526fbe1946p+48`, or `0.0625`, greater than its superset + `0x1.433526fbe1945p+48`; the same-topology union validates exactly + (`test_us_post_transfer_calibration.py:544-753`). + +Weeks remains a valid positive-carrier calibration target. Its source accepts +only integer `-1` or `0..52` and maps `-1` to zero; its QRF path rounds, clips, +positive-UC-gates, and revalidates `0..52`; its carrier event is `weeks > 0` +(`weeks_unemployed.py:791-800,911-983,1218-1222`). Post-transfer amount mapping +uses only positive reference-donor support (`post_transfer_calibration.py:577-626,690-705`). +Count-valued support therefore does not invalidate weighted carrier capacity. + +The ACS runtime's explicit discrete-numeric set contains only two mortgage-year +targets; other numeric targets use the ordinary numeric encoding +(`acs_transfer.py:129-138,3035-3117`). QRF's at-most-32-value “near-discrete” +branch is a leaf-storage optimization, not a carrier semantic type +(`microcosm-fit/qrf.py:388-401,482-503`). Annual child-support and disability +amounts also entered that optimization in the host log, which independently +rules out treating it as a weeks-specific count exception. + +## Seven-target late-transfer audit + +The immutable registry declares seven late targets. Six use +`match_reference`; disability benefits uses `preserve_recipient` and never +emits capacity or prefix evidence +(`post_transfer_calibration.py:208-258,840-932,1319-1335`). + +| Late target | Evidence and verdict | +| --- | --- | +| `child_support_expense` | Covered. Nonnegative annual `CHSP_VAL`, not a count (`child_support.py:166-201,369-383`). Its pinned checkpoint fails the old whole-capacity relationship and validates the final union mechanism. | +| `child_support_received` | Covered. Nonnegative annual `CSP_VAL`, not a count (`child_support.py:166-201,369-383`). Its pinned checkpoint has the same red/green proof. | +| `disability_benefits` | Inapplicable to this capacity bug. It is a nonnegative annual two-slot amount excluding workers' compensation (`disability_benefits.py:184-220,382-395`) and uses `preserve_recipient`; its inspected checkpoint keeps before/after carrier mass at `42,658.57948297383` with `capacity=None` and `selection=None`, as required by the preserve-mode receipt branch (`post_transfer_calibration.py:1319-1335`). | +| `weeks_unemployed` | Covered. Sole semantic count target, integer `0..52`, with carrier additions constrained to positive-UC mutable rows (`weeks_unemployed.py:791-800,911-983,1218-1222`; `stacked_spine.py:8995-9008`). Exact pinned red/green replay proves reducer order caused the failure. | +| `workers_compensation` | Covered. Nonnegative annual `WC_VAL`, not a count (`workers_compensation.py:143-184,337-355`). It uses the default mutable carrier/addition masks (`post_transfer_calibration.py:786-812`) and the shared six-spec regressions. | +| `spm_unit_energy_subsidy` | Covered. Nonnegative measured `SPM_ENGVAL`, checked within unit and reduced to SPM-unit float64 (`energy_subsidy.py:157-233,537-557`). Its entity grain changes the weights, not the set/reduction mechanism; the shared regressions cover its declaration. | +| `pre_subsidy_care_expenses` | Covered. Nonnegative monetary care expense. ACS reconciliation restricts carriers to qualifying people and at most one per tax unit; the late owner admits one stable zero candidate per empty unit (`acs_transfer.py:660-739,1277-1299`; `stacked_spine.py:8728-8746,8977-8986`). The proper-subset six-spec regression covers this constrained structure. | + +Current zero-based late-DAG positions are child support 24, disability 25, +weeks 30, workers' compensation 31, energy subsidy 32, and adult care 34. +Registry scheduling and stacked execution are deterministic, and each group +calibrates before returning (`us_late_producer_registry.py:1338-1396,2013-2019`; +`stacked_spine.py:10054-10095,10927-10931`). The failed host run produced child, +disability, and weeks checkpoints but stopped before workers, energy, and adult +care. Verdicts for those later targets are therefore source/mask proofs plus +shared-kernel regressions, not claims of nonexistent checkpoint replay. ## Verification -The requested suite ran with `UV_NO_SYNC=1` to use the already-synced workspace -environment in the network-restricted sandbox: +On the exact tree committed as `a932974f`, these commands ran serially under +the owner-provided memory guard and exited zero: ```text -uv run --package microcosm-build --extra us --group dev python -m pytest packages/microcosm-data/tests packages/microcosm-build/tests/test_us_fiscal_refresh_builder.py packages/microcosm-build/tests/test_us_state_files_scorer.py -q -264 passed, 3 skipped (267 collected) +uv run pytest packages/microcosm-fit/tests -q +uv run pytest packages/microcosm-calibrate/tests -q +uv run pytest packages/microcosm-data/tests -q +uv run pytest packages/microcosm-frame/tests -q +uv run pytest packages/microcosm-build/tests -q +uv run ruff check . +uv run ruff format --check +git diff --check ``` -Additional receipts: - -- Complete `test_gates.py`: passed. -- Required multiplier grep: zero Python hits. -- Ruff check: clean on all ten touched Python files. -- Ruff format check: clean on the eight non-exempt touched Python files; the - two historical experiment files were not reformatted, as instructed. -- `git diff --check`: clean. -- The medical adjudication block compares byte-for-byte equal to pre-fix - commit `068854d`. -- Pytest emitted non-failing macOS temporary-directory cleanup warnings; no - test failed. - -## Remediation commits - -- `5077f95` — start microcosm#462 Sol remediation progress. -- `c48ba37` — remove the microcosm#462 loss multiplier per microcosm#492. -- `afa910a` — fix Sol finding 1 selector parity. -- `89f74f4` — fix Sol finding 2 CD classifier parity. -- `77040fb` — fix Sol finding 3 relative-error shape. -- `bad7145` — fix Sol finding 4 behavioral containment. -- `3c96514` — apply the finding-2 classifier's required Ruff formatting. - -Nothing was pushed at the time of this report; the branch was subsequently -pushed and merged as #491 (2026-07-22). - -The sandbox rejected writing -`/Users/maxghenis/PolicyEngine/_reviews/sol-491-fix-out.md` with `Operation not -permitted`; the full completion report is therefore committed here and will be -printed to stdout as the requested fallback. +The focused post-transfer file passed all 47 cases. The complete build root +also covered stacked-spine, late-DAG, multispine pool, H5, pool-tool, terminal +receipt, and owner-mask paths. Only established skips and warnings appeared. + +Both current checkpoint commands exited zero with `--expect valid`. Detached +temporary worktrees proved the hardened red side: weeks against `4cc41652` and +both child receipts against `d7b12bab`. The temporary worktrees were removed. + +No host build ran from this lane. The read-only host log and checkpoint tree +were not modified. The owner-provided untracked `.codex-memory-guard.py` and +`_BUILD-FAILURE-1PCT.txt` remain unchanged. Nothing was pushed. + +## Commit lineage and handoff + +- `b533bc61` — open and commit the progress journal; +- `4cc41652` — add the SHA-pinned weeks reproduction and red regression; +- `d7b12bab` — bind candidate capacity to its ordered selection schedule; +- `47742720` — record the cross-target child-support escalation; +- `a932974f` — generate maximum capacity from the attainable row union, add + the child red/green harness and exact proper-subset regressions, complete the + seven-target audit, and record the green suite; and +- the following documentation-only commit — close `PROGRESS.md` and publish + this report without changing the tested executable tree. + +The next authorized action is the serial host owner's 1% rerun at executable +commit `a932974f`. Certification, publication, and release-chain mutation stay +outside this lane. diff --git a/PROGRESS.md b/PROGRESS.md index c7d8d2d8..881476f0 100644 --- a/PROGRESS.md +++ b/PROGRESS.md @@ -1,3 +1,2848 @@ +# Battery package 3 progress + +## Main/F0 calibration-policy merge — 2026-08-22 02:09Z + +### State + +Implementation and verification are complete on the three-main-merge union: +`e66074ad` merges `b4dfa0e7` with the F0 policy port and anti-rot chain folded +in; `e0947020` merges main at `2aa96795` (#733); and `1fc9055e` merges main at +`055dcfaf` (#740). Commit `87e638bd` closes the three deterministic +merge-union guards exposed by the first complete build run. Coverage reports +41,471/41,471 fields and 40/40 inventory checks. The final focused suite, +both unchanged trade-publication tests, repository-wide Ruff, all five test +shards, and the refreshed wheel boundary are green. The authoritative build +receipt covers all 262 files in 17 fresh processes: 6,248 passed, 39 skipped, +exit 0, peak 10.284 GiB. Every accepted shard receipt is below 15 GiB; the +green single-process build attempt that reached 15.989 GiB was rejected and +replaced by the capped receipt. No host pool build, threshold change, +publication, or PR merge occurred. This report-closing commit is the sole +final-push payload; transport to `origin/battery-pkg3-two-part` is the only +remaining step. Earlier sections of this journal are historical records of +their named checkpoints, not current instructions or branch state. + +### Done + +- Re-read `CLAUDE.md` and confirmed the five-shard test, Ruff, no-host-build, + PR-CI/certification, and root-journal boundaries. +- Inspected the clean tracked starting revision `c22e5d37` and confirmed the + local `origin/main` revision is `b4dfa0e7`. +- Confirmed `FINAL_REPORT.md` is the existing report output and that the only + permitted push is the final update to `origin/battery-pkg3-two-part` after + every required check is green. +- Started independent read-only investigations of the F0 schema variant + pattern, PR #741's anti-rot chain, and the two sides' merge intents. +- Audited salvage commit `ea21353d` against its parent, `origin/main`, and the + live merge. It contains the automatic merge plus literal conflict markers, + but no policy-port or anti-rot implementation. Retained the live automatic + merge (including both pool-tool intents); discarded the salvage-only tracked + copies of the owner-untracked memory guard/failure transcript, its conflict + marker snapshots, and the obsolete lineage YAML instead of cherry-picking it. +- Unioned and dated the branch and main root journals, and combined main's F0 + producer-registry test with the branch's policy-identity test at the new + generated-bundle fixture boundary. +- Recovered the third salvage (`abcf7dc1`) in place: staged merge resolution, + closed `regime_gated_qrf_model` schema variant, authored + `post_draw_calibration` declaration plus generator, ninth authority + component at binding version 11, and the walked anti-rot literals + (+92 fields at every count site; `imputation_models` claim 4 → 96). + Discarded the two salvage-only debris files. +- Re-cut the loader semantic-hash golden vector and verified the UK + gate-battery contract mirrors are schema-registry-independent (all ten pin + tests green unchanged). +- 2026-08-22 (this continuation): greened the affected suite on the recovered + tree — 83 tests across the nine spec-engine/pin/lineage/gate-battery files + passed in one process, exit 0. Bisected the spec-identity movement + mechanism by reverting each unstaged schema/semantics/tool edit one at a + time against a synthetic-bundle probe: none moved the golden value, proving + the mover is the branch's staged edits to seed-attested kernel modules + (`spec_engine/seeds.py` digests `us_runtime.acs_transfer` and peers into the + seed-protocol wire inside the spec envelope), not the shared-schema edit. + Corrected the earlier lane-notes claim accordingly. +- 2026-08-22 (headless continuation): re-verified the recovered tree before + committing — 339 tests across the nine affected + spec-engine/pin/lineage/gate-battery/contract files passed in one process, + exit 0 — then committed the staged `b4dfa0e7` merge with the working-tree + F0 port, the re-cut loader golden vector, and the untracked changelog + fragment folded in, as merge commit `e66074ad`. +- Merged `origin/main` at `2aa96795`. Sole conflict: the UK bundle + `spec_sha256` pin in `test_spec_engine_country_bundles.py` (this branch's + `aa32c4c9…` vs main's `e12a2cb8…`, each computed without the other side's + envelope movers). Recomputed all three pins on the union tree via + `load_bundle`: BE `bf022118…` and US `d3de6760…` hold their committed + values; UK moves to `bb711069…`. Verified + `tools/spec_engine_coverage.py --check` (41,471/41,471 fields, 40/40 + inventory checks, no drift) and the same 339-test affected suite green, + then committed merge `e0947020`. +- Ran repository-wide `uv run ruff check .` and `git diff --check` on the + union tree: both pass. +- Walked the CI wheel gate locally because the merge moves + `microcosm-build` packaging and authored spec data: built all five shard + wheels, installed them into a clean venv under the exported lock + constraints, proved the import boundary (all five `microcosm.*` shards + import from the venv prefix; `policyengine_us` absent), ran + `tools/spec_envelope_digests.py be uk` from the installed wheels, and + recomputed `load_bundle` identities from the wheels: BE `bf022118…` and + UK `bb711069…` match the repo pins byte-for-byte. +- Full five-shard suite, one pytest process per shard as in CI: fit exit 0 + (0.56 GiB peak RSS), calibrate exit 0 (0.45 GiB), frame exit 0 + (6.09 GiB), data exit 0 (11.05 GiB). +- Ran the complete build shard to 100%. It exposed three deterministic merge + omissions: direct dataclass walking bypassed the branch's empty + `target_regimes` filter; main's authored-SHA audit did not classify the new + policy-identity digest; and the runtime-graph cardinality pin remained 65 + after `post_transfer_calibration.py` became the 66th reachable module. +- Combined both serializer intents by filtering the opt-in field in the + immutable-safe dataclass path, made the SHA audit allow only the exact + `models/regime_gated_qrf/post_draw_calibration/sha256` identity path in + addition to two external-asset pins, and re-pinned the classified runtime + graph at 66. The three exact failing tests, touched-file Ruff, formatting, + and whitespace checks pass. +- Isolated the build run's other two failures unchanged. Both time out before + publication logic while a child imports `build_us_import_entry_margins.py`; + a constrained import-only probe took 73.82 seconds under load 79, proving + the fixed 60-second failures are host contention rather than a code result. + No timeout, gate, comparator, or threshold was changed. +- Detected that the shared `origin/main` ref advanced six commits while the + build shard ran, then merged `055dcfaf` (#740). The sole conflict was the UK + country-bundle pin (this branch's `bb711069…` vs main's `1f163cbf…`). Fresh + `load_bundle` calls on the union produce BE `bf022118…`, UK `8bf62b6e…`, and + US `d3de6760…`; resolved the pin with the third union-only value. +- On the newest union, `tools/spec_engine_coverage.py --check` reports + 41,471/41,471 configuration fields and 40/40 inventory checks. The focused + policy identity, country bundle, field usage, coverage, inventory, + gate-battery/data-contract, serializer, authored-SHA, and spine-blind import + graph suite passes in one process. +- Re-ran the two unchanged trade-publication crash tests under lower host load; + both pass with their existing 60-second subprocess bound. The earlier + failures were import-time host contention, not publication behavior, and no + timeout or product threshold changed. +- Re-ran repository-wide Ruff and whitespace checks on the newest union: both + pass. +- Completed the final five-shard suite with measured RSS. Fit: 93 passed, + 0.862 GiB. Calibrate: 203 passed, 0.449 GiB. Frame: 294 passed / 36 skipped, + 6.492 GiB. Data: 275 passed / one skipped, 11.052 GiB. A green build-root + run reached 15.989 GiB and was not accepted; the authoritative rerun covered + every one of its 262 test files across 17 fresh pytest processes, reproduced + 6,248 passed / 39 skipped with exit 0, and peaked at 10.284 GiB under a + 12 GiB guard without intervention. +- Refreshed all five wheels on the newest union. Reinstalled into the existing + clean, lock-constrained wheel venv, every namespace imports from the venv + prefix with `policyengine_us` absent; installed-wheel bundle identities are + BE `bf022118…`, UK `8bf62b6e…`, and US `d3de6760…`. A newly created offline + venv stopped at uncached third-party resolution, so no network-dependent + install claim is made. +- Wrote the completed r4 handoff and exact verification receipts to + `FINAL_REPORT.md`. + +### Next + +1. Push this report-closing commit once to + `origin/battery-pkg3-two-part`; PR #742 then updates in place. Do not merge + the PR. No implementation or verification work remains. + +## Post-transfer receipt validation — 2026-08-21 14:48Z + +### State + +Complete as of 2026-08-21 20:14Z. Executable commit `a932974f` binds each +candidate endpoint to its selection schedule and derives the whole attainable +ceiling from one row union using the recipient total's reduction topology. +Both exact validator inequalities remain unchanged. Red and green checkpoint +replays, all 47 focused receipt tests, all five package test roots, repository- +wide Ruff, touched-file formatting, and whitespace checks pass. The final +report is written to `FINAL_REPORT.md`; this closing commit is documentation- +only. The serial host queue remains the only owner of the next 1% build. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed `FINAL_REPORT.md` is the required output file and preserved the + owner-provided untracked memory guard and host-failure transcript unchanged. +- Confirmed the normal GitNexus query/context interface is unavailable in this + session; direct checkpoint, source, call-site, and history tracing is the + documented fallback. +- Started independent read-only checkpoint-reproduction, mechanism/audit, and + regression/history investigations. None may edit or commit. +- Reopened and updated this committed state/done/next journal before any + executable or test change. +- Ran `uv run pytest` separately for all five package test roots + (`microcosm-frame`, `microcosm-fit`, `microcosm-calibrate`, + `microcosm-data`, and `microcosm-build`) under the owner-provided 12 GiB + guard; every command exited zero with only expected skips/warnings. +- Ran repository-wide `ruff check .` and `git diff --check`; both passed. +- Replayed the preserved assembled, unemployment-compensation, and + weeks-unemployed checkpoints without invoking any build. The replay reaches + the exact supplied validator error with 4,311 ASEC reference rows, 34,293 + ACS recipient/mutable rows, and 32 positive-UC addition candidates. +- Isolated the only false receipt relationship: the ID-ordered prefix reports + `85,676.23791782456`, while the separately masked candidate-capacity sum is + `85,676.23791782455`. The exact `upper_prefix_mass <= + addition_candidate_mass` proof therefore fails by one float64 ULP + (`1.4551915228366852e-11`); every other relationship passes. +- Confirmed this is not a count-target exception. The weeks source and its QRF + postprocessing constrain values to integer `0..52`, post-transfer amount + mapping constrains mutable values to positive reference-donor support, and + the carrier event remains `weeks > 0`; the defect is two reductions for one + declared ordered capacity. At reproduction commit + `4cc41652`, the two paths are visible at + `post_transfer_calibration.py:463-493,817-891`, and the unchanged exact + relationship at `post_transfer_calibration.py:1389-1490` + (`weeks_unemployed.py:791-800,911-983,1218-1222`; + `post_transfer_calibration.py:588-626,702-705`). +- Added `tools/reproduce_us_post_transfer_weeks_checkpoint.py`, which validates + all three file digests plus both target-checkpoint identity/raw-draw digests, + restores only the native clone-0 calibration vectors, and reports the exact + error and failed predicate + without executing a fit or build (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:1-6,74-112,115-250`). +- Ran that harness against the read-only SHA-addressed host tree with + `--expect invalid`; it exited zero in 7.4 seconds and reported only + `upper_prefix_mass <= addition_candidate_mass` as false, with the exact + `1.4551915228366852e-11` delta. +- Added a hermetic expected-failure regression containing the exact 32 + production candidate-weight bytes (SHA-256 `a32b75bc...aa5a8`) plus a + nonmutable recipient ballast row. The full post-transfer test file passes + 29/29 before the fix + (`packages/microcosm-build/tests/test_us_post_transfer_calibration.py:537-631`). +- Implemented one immutable `_PrefixSchedule` for each declared carrier order. + Capacity and nearest-prefix selection now consume the same cumulative + float64 path for both removal and addition; the strict receipt comparator is + byte-for-byte unchanged + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:282-287,445-515,839-922,1471-1520`). +- Replayed the SHA-pinned checkpoint with `--expect valid`; it exited zero in + 7.0 seconds. Candidate capacity and upper prefix are both + `85,676.23791782456`, their delta is zero, row selection is unchanged, and + strict receipt validation succeeds. +- Converted the production-weight regression to require strict validation for + every late `match_reference` spec and added the symmetric removal-path + regression (`packages/microcosm-build/tests/test_us_post_transfer_calibration.py:537-694`). +- Ran all 35 post-transfer calibration tests and the affected stacked-spine, + multispine-pool, pool-H5, and pool-tool matrix; every test passed. Touched + Python files pass formatting and Ruff. +- Audited all seven late targets. The shared schedule applies to the six + `match_reference` targets, including the sole count target + `weeks_unemployed`; `disability_benefits` preserves recipient carriers and + never enters the capacity/prefix branch. Detailed source-cited verdicts are + in `_LANE-NOTES.md`. +- Ran `uv run pytest` separately for the final repaired tree's + `microcosm-frame`, `microcosm-fit`, `microcosm-calibrate`, `microcosm-data`, + and `microcosm-build` test roots under the 12 GiB/20 ms guard. Every command + exited zero; only the established skips and warnings appeared. +- Ran repository-wide `ruff check .`, touched-file `ruff format --check`, and + `git diff --check`; all passed. +- Committed the shared ordered-prefix generating repair, strict regressions, + complete late-target audit, and green verification record at `d7b12bab`. +- Replayed both preserved child-support target checkpoints against + `d7b12bab`. Each now has a valid candidate/prefix relationship but fails the + distinct exact `maximum_attainable_mass <= recipient_total` relationship. + Recipient total is `79,926,522.10879111`; both partition-composed maxima are + `79,926,522.10879174`, an excess of `6.258487701416016e-07`. +- Isolated the two child-support partitions. Expense has allowed-positive mass + `71,696.09739141785` and addition mass `79,854,826.01140033`; received has + allowed-positive mass `180,209.75664861224` and addition mass + `79,746,312.35214312`. Both sum to the same invalid maximum. This is the same + generating class at the whole-capacity partition level, not evidence for a + looser validator (`post_transfer_calibration.py:824-875,1456-1466`). +- Added `tools/audit_us_post_transfer_child_support_checkpoints.py`. It pins the + assembled Frame and both child target file/identity/raw-draw hashes, + reconstructs production clone-0 values and half weights, and requires both + targets to match the exact expected relationship, ValueError, and float + values on either the red or green side. +- Replayed that harness against `d7b12bab` with `--expect invalid`. Both targets + fail only `maximum_attainable_mass <= recipient_total`; the maximum is + `79,926,522.10879174`, recipient total is `79,926,522.10879111`, and the + delta is `6.258487701416016e-07`. +- Replaced the partition-scalar maximum with a single attainable-row union + zero-masked onto the recipient-weight vector. Its identical length, order, + and nonnegative reduction topology makes the exact maximum/subset bound + structural without a clamp, tolerance, or validator change + (`post_transfer_calibration.py:823-885,1457-1529`). +- Added two six-spec regressions: one reproduces the child partition sum one + ULP above the whole recipient reduction; the other is a constrained proper + subset whose regrouped sum is `0.0625` above its superset. Both now validate, + along with the production weeks and symmetric removal fixtures + (`test_us_post_transfer_calibration.py:544-753`). +- Replayed the pinned weeks failure against `4cc41652` and the child failures + against `d7b12bab`; the hardened current harnesses accepted only their exact + respective predicates and numbers. Against the current tree, weeks and both + child receipts validate. Child maximum equals recipient total exactly while + the historical partition endpoint remains 42 ULP higher; weeks candidate and + upper prefix both equal `85,676.23791782456`. +- Audited every late target again. Six `match_reference` declarations share the + repaired kernel and are enumerated by the regressions; disability uses + `preserve_recipient` and has no capacity proof. The source- and checkpoint- + qualified verdict for each target is in `_LANE-NOTES.md`. +- Ran all 47 focused post-transfer tests; all passed under the memory guard. +- Ran `uv run pytest` separately for `microcosm-fit`, + `microcosm-calibrate`, `microcosm-data`, `microcosm-frame`, and the complete + `microcosm-build` test roots on the final executable tree. Every command + exited zero; only established skips and warnings appeared. +- Ran repository-wide `ruff check .`, touched-file `ruff format --check`, and + `git diff --check`; all passed. +- Committed the complete executable repair, pinned harnesses, strict + regressions, seven-target audit, and green verification record at + `a932974f`. +- Rewrote `FINAL_REPORT.md` with the exact weeks and child red/green evidence, + semantic adjudication, target-by-target verdicts, verification boundary, + commit lineage, and host-owner handoff. + +### Next + +1. The serial host owner reruns the restricted 1% build at executable commit + `a932974f`. +2. Certification, publication, and release-chain mutation remain with their + existing owners. + +## Owner continuation revalidation — 2026-08-21 11:45Z + +### State + +In progress. The supplied host traceback is the active boundary. The branch +already records a scoped executable correction and regression lineage, but this +continuation is independently checking the current raise-site flow, exact +assigned-target selection, regression strength, and current verification state +before accepting that prior result. The untracked owner memory guard and host +failure transcript are preserved unchanged. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed `FINAL_REPORT.md` is the required output file and inspected the + existing correction/report lineage at starting revision `14666f6b`. +- Confirmed normal GitNexus graph-query tools are unavailable in this session; + direct source, call-site, history, and regression tracing is the documented + fallback. +- Reopened this committed progress journal before making any implementation or + test change. +- Reconstructed the historical failure from `33bf52fe`: receipt production + attached calibration-audit QRF evidence to every imputed target, and + `validate_stacked_gap_fill_receipt` called the strict evidence validator + before testing whether a target had an assigned calibration spec. The + physical `puf_tax_itemization__batch_1` record therefore failed binding to + the public `puf_tax_itemization` family at the exact supplied raise site. +- Confirmed the current correction uses four matching assignment fences: + `transfer_acs_inputs` defaults the evidence selection to empty; the ordinary + and banked fits retain regimes only for selected targets; both stacked + producers derive selection from the immutable nine-target registry; and both + terminal validators reject evidence on unassigned targets before entering + strict record binding. Taxable interest is absent from the registry. +- Traced every production `transfer_acs_inputs` caller. Only the canonical + early and late stacked owners pass `regime_evidence_targets`; generic + multispine and pool-tool callers use the empty default. +- Created a detached temporary worktree at failing-first commit `22b2c6bc`, + whose runtime blobs still equal `33bf52fe`, and ran the exact synthetic + taxable-interest regression there. It failed at historical lines 4512 and + 4310 with `ACS QRF pattern record binding is invalid`, matching the supplied + traceback. Removed the temporary worktree afterward. +- Collected and ran the current 16-case boundary matrix: the exact synthetic + regression, ordinary and banked real 15-target transfers, all 12 fully + rehashed strict-binding mutations, and mixed selected/unselected draw + preservation. All 16 passed; the only warning was joblib's physical-core + fallback. +- Reconciled the correction lineage and Git objects. Runtime repair starts at + `176c60fc`, with strict binding/count hardening in `887df056` and `94b7aecb`; + real producer-to-validator coverage closes at `ad2a44c1`. Current build + source and tests are byte-identical to that reviewed checkpoint. +- Read the external host retry without controlling it. At the + `2026-08-21T11:48:32Z` snapshot it had crossed taxable interest, completed + survey target 47/47 plus housing 1/1 without the reported error, and remained + active in the primary-QRF stage. No final pool, manifest, gates, or terminal + revision-bound status existed, so this is progress evidence only. +- Reconciled independent runtime-scope and regression audits. Neither found a + residual canonical leak or a missing boundary case. The only intentionally + generic seam is that a direct noncanonical library caller may explicitly + request regime diagnostics for a target on its own transfer surface; no + production caller or canonical receipt validator promotes that diagnostic to + calibration evidence. No duplicate test or runtime edit is warranted. +- Ran all five directly affected files together under the owner-provided + 12 GiB/20 ms guard. Fresh collection was 64 ordinary-transfer, five + multispine-serialization, 259 stacked-spine, 164 pool-tool, and 38 H5-I/O + tests: all 530 passed in one process with exit zero. Maximum observed + per-process RSS was 1.550 GiB. +- Ran repository-wide `ruff check .`, formatting checks on all 15 Python files + changed since `33bf52fe^`, and committed/current `git diff --check`; all + passed. +- Started the complete repository suite in one guarded process. Pytest reached + 83% with no test failure, then the memory guard terminated the process when + one process reached 13.225 GiB against the 12 GiB ceiling. This is a resource + termination, not a green full-suite result, and is retained explicitly. The + repository suite must be rerun in fresh bounded shards. +- Ran all four sibling packages in fresh guarded processes. `microcosm-fit` + passed 93 tests at 0.785 GiB peak; `microcosm-calibrate` passed 201 at + 0.438 GiB; `microcosm-data` passed 275 with one expected skip at 11.053 GiB; + and `microcosm-frame` passed 294 with 36 expected skips at 6.493 GiB. These + green shards account for all 900 non-build cases. + +### Next + +1. Run all build-package core+UK, US a-r, and US s-z shards on the unchanged + executable tree, preserving the 12 GiB ceiling. +2. Refresh and commit `FINAL_REPORT.md`; leave restricted host certification + and publication outside this local correction. + +## Owner continuation audit — 2026-08-21 10:17Z + +### State + +Complete as of `2026-08-21 11:44:09Z`. The supplied traceback maps to the +historical globally attached QRF audit-evidence path at `33bf52fe`. The current +tree already contains the assigned-only executable correction and the exact +synthetic plus real ordinary/banked regressions, so no duplicate runtime or +test edit was warranted. Focused, directly affected, repository-wide, static, +and Git-object checks are green. `FINAL_REPORT.md` contains the current result +and is committed at `811277bb`. The live restricted-host retry has passed the +reported taxable-interest boundary but remains nonterminal, so host +certification is not claimed. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the branch, recent correction lineage, required output file, and + pre-existing untracked memory-guard and host-failure diagnostics. +- Reopened this committed progress journal before changing runtime or tests. +- Started independent read-only audits of the invariant, calibration diff, and + regression surface. +- Confirmed the repository is not registered in the normal GitNexus CLI index; + direct raise-site and Git-history tracing will remain the fallback unless a + usable preserved index is found. +- Followed the debugging skill's unindexed-repository instruction and attempted + a local GitNexus analysis. Source parsing completed far enough to create a + 100 MiB partial index, but registry registration was blocked by the sandbox + at `/Users/maxghenis/.gitnexus/registry.json`; moved the generated index out + of the worktree to a recoverable `/private/tmp` location. +- Reconstructed the historical failure at exact source lines. The public + 15-target `puf_tax_itemization` family was split at width eight, so taxable + interest legitimately carried physical family + `puf_tax_itemization__batch_1`. Old receipt production attached QRF evidence + to every imputed target, and old validation entered strict public-family + binding before checking assignment, producing the supplied line-4512 to + line-4310 error. +- Confirmed the correct repair is assignment scoping, not accepting batch + aliases. Current strict entity/family/target/count binding remains unchanged + for declared targets. Unassigned targets must retain the four legacy counts + but are rejected if QRF or calibration evidence is attached. +- Traced the current four matching fences: the transfer API defaults its audit + selection to empty; both ordinary and banked fits detect, verify, and retain + regime provenance only for selected targets; the stacked owner and receipt + producer derive the same selection from the immutable calibration registry; + and terminal validation enters QRF binding only for a declared spec. +- Enumerated the immutable surface: two early and seven late model-required, + source-operator, and adult-care targets. `taxable_interest_income` is absent. +- Re-read the exact synthetic taxable-interest `__batch_1` regression, the + real 15-target ordinary/banked producer-to-validator regression, mixed-family + draw-preservation coverage, legacy-count checks, and strict rehashed family + forgery coverage. Together they capture both non-leakage and fail-closed + binding without weakening the invariant. +- Reconciled independent read-only invariant and test/log audits; both reached + the same root cause and found no residual canonical scope leak. The initial + external snapshot was empty and nonterminal; a later retry became active. +- Ran the exact taxable-interest binding regression, both real wide-family + ordinary/banked cases, all 12 rehashed strict-binding mutations, and the + mixed-family draw-preservation regression under the 12 GiB guard. All 16 + checks passed; the only warning was joblib's physical-core fallback. The + first launcher attempt never reached collection because `uv` could not write + its default user cache, so the successful run used a task-local cache under + `/private/tmp`. +- Ran all five directly affected files together under the same guard. Fresh + collection was 64 ordinary-transfer, five multispine, 259 stacked-spine, 164 + pool-tool, and 38 H5 receipt-I/O tests. All 530 completed with exit zero; the + guard reported 0.029 GiB maximum observed per-process RSS. +- Ran the repository-wide pytest suite under the guard. Fresh collection was + 6,609 items; the run reached 100% with expected skips and exit zero. The + guard again reported 0.029 GiB maximum observed per-process RSS. +- Ran repository-wide `ruff check .`, changed-range `ruff format --check` on + all 15 Python files changed since `33bf52fe^`, committed-range whitespace + checks, index/worktree whitespace checks, and final tracked-tree cleanliness + checks; all passed. A diagnostic repository-wide format check identified 49 + pre-existing out-of-range files, so no unrelated bulk reformat was made. +- Bound the current Git objects to earlier reviewed checkpoints. The current + `microcosm-build/src` tree is `7234ac19`, identical to both complete-suite + checkpoint `d29a8705` and reviewed regression checkpoint `ad2a44c1`. The + current build-tests tree is `0c5d7816`, identical to `ad2a44c1`; relative to + `d29a8705`, its only test change is the strengthened ordinary/banked terminal + validator regression. Everything after `ad2a44c1` is journal-only. +- Took the final read-only host snapshot at `2026-08-21 11:42:58Z`. The retry + had written taxable interest as target 22/47 with physical family + `puf_tax_itemization__batch_1` and progressed through target 39/47 without a + traceback, `ValueError`, or binding failure. It still had no runner exit + marker or final pool, manifest, and gates artifacts, and its mutable log did + not bind the launch to a recoverable Microcosm SHA. +- Refreshed and committed the required `FINAL_REPORT.md` at `811277bb` with the + current diagnosis, assigned-only fences, exact regression evidence, fresh + 6,609-item suite, static/object checks, and nonterminal host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing final `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not +publish or mutate the release chain as a side effect of that external +verification. + +## Owner continuation revalidation — 2026-08-21 09:47Z + +### State + +Complete as of `2026-08-21 10:14:50Z`. The supplied traceback is independently +bound to the historical globally enabled QRF-audit evidence path at +`33bf52fe`. The current tree already contains the narrow assigned-only repair +and exact regression coverage, so no duplicate executable or test edit was +warranted. The decisive cases, all 530 directly affected tests, static checks, +and exact prior-suite object bindings are green. `FINAL_REPORT.md` contains the +revision-bound result and is committed at `0ed1053d`. The final read-only host +snapshot is nonterminal, so restricted-build certification is not claimed. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the required `PROGRESS.md` and `FINAL_REPORT.md` are tracked and + committed at the start of this continuation. +- Inspected branch status and the most recent continuation commit; preserved + the pre-existing untracked memory-guard and host-failure diagnostics. +- Confirmed the normal GitNexus query/context tools are not exposed, then used + the preserved graph indexed at `42819d53` through GitNexus's read-only local + backend. Error-text query and context found the exact synthetic regression; + custom Cypher proved the oversized production module was omitted, so direct + source and Git-object tracing supplied the production path. +- Reconstructed the failure at `33bf52fe`: the public 15-target + `puf_tax_itemization` family was split at width eight, taxable interest's + legitimate physical record was `puf_tax_itemization__batch_1`, the producer + attached QRF audit regimes/evidence to every target, and old line 4512 invoked + strict binding before assignment lookup. Old line 4310 therefore compared + the physical batch with the public family and raised the supplied error. +- Confirmed that accepting a batch alias would weaken the invariant without + fixing the leak. Current validators retain exact entity/family/target binding + for declared targets, but reject QRF or calibration evidence on an undeclared + target before entering that binding path. +- Traced the current selection through `transfer_acs_inputs`, ordinary and + banked fits, per-target provenance, both receipt producers, both canonical + stacked callers, both terminal validators, and calibration application. + Generic callers default to no calibration-audit regime work; only the two + stacked owners select immutable registry specs; unselected targets keep + ordinary draws and empty audit regimes and receive neither evidence nor + calibration writes. +- Enumerated the immutable policy: two early and seven late model-required, + source-operator, and adult-care targets. `taxable_interest_income` is absent. +- Re-read the exact synthetic regression, the real 15-target ordinary/banked + producer-to-validator regression, mixed-family draw-preservation coverage, + legacy four-count checks, and the fully rehashed strict-family forgery case. + The requested executable and regression changes are already committed, so a + duplicate source or test edit is not warranted. +- Reconciled three independent read-only audits of runtime reachability, + regression strength, and correction history. They agree that no canonical + scope leak remains and that exact assigned-target family binding is still + fail-closed. The history audit bound the bad executable blob to `33bf52fe` + but correctly notes that the overwritten external log cannot prove its + process-launch journal SHA. +- Recorded the important scope precision: every fitted target still uses the + QRF's unchanged operational regime logic. What is selected-only is the + additional calibration audit detection and verification, per-target regime + provenance, QRF receipt evidence, and post-transfer calibration writes. +- Ran the exact synthetic taxable-interest validator case plus the real + ordinary and banked 15-target producer-to-validator cases under the owner's + 12 GiB/20 ms guard. All three passed; peak observed per-process RSS was + 0.571 GiB and the only warning was joblib's physical-core fallback. +- Ran all 12 fully rehashed QRF structure mutations separately. All passed + under the same guard with 0.389 GiB maximum observed per-process RSS. +- Ran the five directly affected files together under the guard. Collection + was 64 ordinary-transfer, five multispine, 259 stacked-spine, 164 pool-tool, + and 38 H5 receipt-I/O tests. All 530 passed with exit zero and 1.602 GiB + maximum observed per-process RSS. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, `git diff --check 33bf52fe^..HEAD`, and + worktree whitespace checks. All passed. +- Bound the current Git objects to prior verification checkpoints. The current + `microcosm-build/src` tree is `7234ac19`, identical to both reviewed + regression checkpoint `ad2a44c1` and complete-suite checkpoint `d29a8705`; + current build source/tests are byte-identical to `ad2a44c1`, and current + production source, tools, specs, project, and lock objects are byte-identical + to `d29a8705`. +- Took the final read-only host snapshot at `2026-08-21 10:13:00Z`. The external + directory contained only `build.log` and `guard.log`; `build.log` had been + truncated to zero bytes at `10:07:12Z`, while the latest guard heartbeat at + `10:09:11Z` still reported a resource wait behind one other job. There was no + traceback to inspect, but also no runner exit marker, transferred/simulated + checkpoint, `pool.h5`, `pool.manifest.json`, or `pool.gates.json`. This is no + terminal host verdict and does not alter the local result. +- Refreshed and committed the required `FINAL_REPORT.md` at `0ed1053d` with the + corrected operational-versus-audit regime distinction, root cause, + assigned-only fences, regression evidence, fresh guarded verification, + static/object checks, and explicit nonterminal host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or +mutate the release chain as a side effect of that external verification. + +## Owner continuation verification — 2026-08-21 09:17Z + +### State + +Complete as of `2026-08-21 09:45:41Z`. The owner traceback binds exactly to the +historical globally enabled QRF-evidence path at `33bf52fe`. +The current tree already contains the narrow assigned-only correction and the +exact synthetic plus real ordinary/banked regressions. Three independent +read-only audits found no remaining canonical path by which taxable interest +can receive regime evidence, a QRF receipt, or a calibration write, so no +duplicate runtime or test edit is warranted. All 530 directly affected tests, +static checks, and exact prior-suite object bindings are green. The required +`FINAL_REPORT.md` is refreshed and committed. The externally owned host retry +remains live and nonterminal, so host certification is not claimed. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed `PROGRESS.md` and the required `FINAL_REPORT.md` are tracked, and + preserved the pre-existing untracked memory-guard and host-failure artifacts. +- Inspected branch status, current journal/report state, and recent correction + lineage without treating the earlier completion verdict as current proof. +- Confirmed GitNexus graph query/context tools are unavailable in this session; + the required workflow will use its CLI/index route where possible and direct + source/history tracing as the documented fallback. +- Used the preserved GitNexus graph indexed at `42819d53` through its read-only + local backend. Error-text query found the stacked regression surface; context + resolved the exact synthetic regression, while custom Cypher confirmed the + oversized production module was omitted from the graph. Direct source and + Git-object tracing therefore supplied the missing production call path. +- Reconstructed the historical failure at `33bf52fe`. The 15-target public + `puf_tax_itemization` family was physically split at width eight, so taxable + interest carried `puf_tax_itemization__batch_1`. The old producer attached + QRF evidence to every imputed target and old line 4512 invoked strict record + binding before consulting assignment; old line 4310 compared the physical + batch name with the public family and raised the supplied error. +- Confirmed the current correction keeps ordinary QRF imputation intact while + scoping only calibration-specific work. `transfer_acs_inputs` defaults its + regime selection to empty; the two canonical stacked callers derive exact + selections from the immutable registry; ordinary and banked fits detect and + verify only selected regimes; unselected records carry empty regimes; and + receipt producers attach evidence only to the same selected set. +- Enumerated the live policy and selectors: exactly two early and seven late + model-required, source-operator, and adult-care specs are selected. + `taxable_interest_income` is absent. Both terminal validators validate the + four legacy counts for every target, reject QRF/calibration evidence on an + undeclared target, and run unchanged exact entity/family/target binding only + after a declared spec is found. +- Audited all four production `transfer_acs_inputs` callers. Generic + multispine and pool-tool paths retain the empty default; only canonical early + and late stacked owners opt into the registry-derived selection. A generic + library caller may intentionally request regime metadata for any target on + its own transfer surface, but no production caller exposes that choice and + canonical validators reject undeclared evidence. +- Reconciled independent runtime, history, and regression audits. They found no + remaining canonical scope leak. The exact synthetic regression rejects + undeclared taxable-interest evidence then accepts its evidence-free + four-count receipt; the real 15-target ordinary/banked regression preserves + the `__batch_1` record, strips its regimes/evidence, retains selected + unemployment evidence, and sends the generated taxable receipt through the + terminal validator. Rehashed mutations separately prove strict assigned + record binding remains fail-closed. +- Confirmed the historical regression commit `22b2c6bc` retained the exact + `33bf52fe` runtime objects, while the current production source tree is + byte-identical to the complete-suite checkpoint `d29a8705` and the current + production/test trees are byte-identical to reviewed checkpoint `ad2a44c1`. +- An independent focused run passed the synthetic plus real ordinary/banked + cases (3 passed), and a separate strict-binding mutation run passed all 12 + parameter variants. These used the project environment without the owner + memory guard; the guarded affected-suite verification below independently + supersedes that resource boundary. +- Ran all five directly affected files together under the owner-provided 12 GiB + and 20 ms guard. Collection was 64 ordinary-transfer, five generic + multispine, 259 stacked-spine, 164 pool-tool, and 38 H5 receipt-I/O tests. + All 530 passed in one process with exit zero and 1.596 GiB maximum observed + per-process RSS; warning display was disabled for this broad run. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, `git diff --check 33bf52fe^..HEAD`, and + worktree whitespace checks. All passed. +- Reconfirmed exact Git tree identities: current `microcosm-build/src`, + `tools`, and `specs` match complete-suite checkpoint `d29a8705`; current + production and tests match reviewed regression checkpoint `ad2a44c1`. The + only package-test difference from the complete-suite checkpoint is the + strengthened stacked-spine regression file, whose complete 259-test run is + included in the fresh affected matrix. +- Reviewed the two pre-existing SIPP streaming-inference changes flagged by + the history audit. Their vehicle and voluntary-filing outputs do not overlap + any of the nine calibration targets or their call paths. They are documented + in `_LANE-NOTES.md` as independently locked-fact-tested memory mitigations, + so this continuation preserves them rather than expanding into an unrelated + parser revert. +- Refreshed and committed the required `FINAL_REPORT.md` at `99d1279b` with the + current diagnosis, assigned-only correction, regression evidence, fresh + guarded matrix, static/object checks, branch-scope qualifications, and exact + remaining host boundary. +- Took the final read-only host snapshot at `2026-08-21 09:45:41Z`. The retry + had rebuilt taxable interest and progressed into + `person/puf_tax_itemization__batch_3` without the reported exception. The log + remained active, but the directory still contained only `build.log` and + `guard.log`; there was no traceback, binding error, terminal marker, + transferred/simulated checkpoint, or final pool/manifest/gates artifact. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or +mutate the release chain as a side effect of that external verification. + +## Owner continuation recheck — 2026-08-21 08:58Z + +### State + +Complete as of `2026-08-21 09:16:03Z`. The current committed assigned-only +correction and regressions withstand an independent source/history audit, the +focused cases, all 530 directly affected tests, static checks, and exact +production-object comparison. No duplicate executable or test edit was +warranted. The required final report is refreshed with the latest evidence. +The restricted host retry remains live and has not recurred at the reported +failure, but it has not reached terminal stacked validation or emitted final +artifacts, so host certification is not claimed. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed `PROGRESS.md` and the required `FINAL_REPORT.md` are tracked and + preserved the two pre-existing untracked diagnostic artifacts. +- Inspected branch status, recent commit lineage, and the prior report without + assuming their completion verdict remains current. +- Confirmed the GitNexus MCP query/context tools are unavailable in this + session; direct raise-site, caller, history, and focused-test tracing is the + documented fallback. +- Re-read the current transfer selection, per-target provenance stripping, + early validator fence, and both taxable-interest regressions at their exact + source locations. The validator checks legacy counts for every target but + rejects QRF/calibration evidence before strict record binding when the target + is outside the immutable early calibration registry. +- Ran the exact unassigned taxable-interest binding regression and both real + 15-target producer cases (ordinary and banked) under the owner 12 GiB guard. + All three passed with exit zero and 0.566 GiB maximum observed per-process + RSS; output was limited to the known joblib physical-core fallback warning. +- Reconciled independent read-only source and regression/history audits. Both + confirmed there is no remaining canonical leak and recommended no duplicate + executable or test change. The real ordinary/banked regression uses all 15 + public-family targets, retains taxable interest's physical `__batch_1` + record, proves selected unemployment evidence remains non-vacuously present, + and sends the generated taxable receipt through the terminal validator. +- Confirmed exact family binding remains fail-closed for assigned targets; a + separate fully rehashed mutation test rejects a physical batch alias rather + than weakening the invariant that exposed the historical leak. +- Ran all five directly affected files together under the owner guard: 64 + ordinary-transfer tests, five multispine-serialization tests, 259 + stacked-spine tests, 164 pool-tool tests, and 38 H5 receipt-I/O tests. All + 530 passed with exit zero and 1.633 GiB maximum observed per-process RSS. +- Received a read-only host snapshot at `2026-08-21 09:00:39Z`. The retry was + still live with no traceback or binding error, had passed taxable interest + and completed late-producer batch 4, but had no terminal exit marker or final + pool/manifest/gates artifacts. Host certification therefore remains open. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range whitespace checks, and worktree + whitespace checks; all passed. +- Confirmed the current production source, tools, specs, project, and lock + objects exactly match complete-suite checkpoint `d29a8705`, while the current + production and test objects exactly match reviewed regression checkpoint + `ad2a44c1`. No executable or regression edit is warranted in this recheck. +- Rechecked the host at `2026-08-21 09:12:27Z`. The retry was still live and + had completed late-producer batch 5 target 4/5 without any traceback, + `ValueError`, or binding text; the final boundary remained unreached. +- Took the final read-only host snapshot at `2026-08-21 09:15:36Z`. Taxable + interest remained successfully checkpointed, the retry had entered the next + late-producer DAG, and live writers and guard heartbeats remained present. + There was still no traceback, `ValueError`, binding text, exit marker, + transferred/simulated checkpoint, or final pool/manifest/gates artifact. +- Refreshed the required `FINAL_REPORT.md` with the current focused and + affected-suite results, static/object checks, independent audit verdicts, + and precise provisional host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing final `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not +publish or mutate the release chain as a side effect of that external check. + +## Owner continuation audit — 2026-08-21 08:35Z + +### State + +Complete as of `2026-08-21 08:56:50Z`. The supplied traceback is bound to the +historical globally enabled QRF evidence path, while the current tip contains +the narrow assigned-only correction, the exact taxable-interest regression, +and the real ordinary/banked producer-to-validator regression. Independent +source, history, and test audits found no remaining canonical leak, so this +continuation made no duplicate runtime or test edit. The focused matrix, all +530 directly affected tests, static checks, and production-object comparison +are green. The required final report is committed at `dc4055c5`; restricted +host verification remains live and nonterminal. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Preserved the untracked owner-provided failure and memory-guard artifacts. +- Confirmed `FINAL_REPORT.md` is the required output file and that the existing + progress and report journals are committed at the start of this continuation. +- Inspected the current branch status, recent correction lineage, assigned lane + notes, current report, and prior verification record. +- Started independent read-only audits of the invariant path, branch charter, + and regression surface. +- Reconstructed the supplied failure at `33bf52fe`: the 15-target + `puf_tax_itemization` family produced the legitimate bounded physical family + `puf_tax_itemization__batch_1`, but QRF evidence was attached to unassigned + taxable interest and terminal validation invoked strict binding before the + assignment lookup. The unsplit-family comparison therefore raised at the + cited historical lines 4512 and 4310. +- Confirmed accepting a batch alias would be unsafe and incomplete: the next + historical check expected all 15 public-family targets while the physical + record contained only its bounded batch. The correct fix is to prevent + calibration-specific regime work and evidence outside the assigned surface. +- Traced the current assigned-only fences through ordinary and banked fitting, + per-record provenance, early and late receipt construction, and both + terminal validators. Unassigned targets retain the required four legacy + transfer counts, reject forged QRF/calibration evidence, and bypass strict + calibration record binding; selected targets retain exact binding. +- Reconciled three independent read-only audits. All identified taxable + interest as out of scope and found the current correction and regression + coverage appropriately narrow. +- Attempted the GitNexus query/context workflow. A fresh local graph indexed + current commit `42819d5`, but sandbox denial of + `~/.gitnexus/registry.json` prevented CLI query registration. The generated + index was moved intact to + `/private/tmp/microcosm-pkg3-two-part-gitnexus-42819d53`; direct source and + Git-object tracing supplied the fallback evidence. +- Ran the exact unassigned taxable-interest validator regression and the real + ordinary/banked 15-target producer-to-validator regression under the owner + 12 GiB/20 ms guard. All three cases passed; output was limited to the known + joblib physical-core fallback warning. +- Ran all five directly affected files together under the owner guard: 64 + ordinary-transfer tests, five multispine-serialization tests, 259 + stacked-spine tests, 164 pool-tool tests, and 38 H5 receipt-I/O tests. All + 530 passed with exit zero; maximum observed per-process RSS was 1.659 GiB. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 extant Python + files changed since `33bf52fe^`, and `git diff --check + 33bf52fe^..d67df37c`; all passed. The sandbox-blocked user UV cache was + bypassed with the already-synchronized project virtual environment. +- Confirmed all production package source trees, `tools`, `specs`, + `pyproject.toml`, and `uv.lock` are exact Git-object matches to complete-suite + checkpoint `d29a8705`. Only the journals and the strengthened stacked-spine + regression differ. +- Took a read-only restricted-host snapshot at `2026-08-21 08:42:44Z`. The + retry was live and had completed the taxable-interest fit without an error, + but it had not yet reached the final stacked gap-fill receipt validator. No + terminal exit marker, final pool H5, manifest, or gates artifact existed, so + neither passage of the supplied invariant nor host certification is claimed. +- Rechecked the host at `2026-08-21 08:53:11Z`. All eight late + `puf_tax_itemization__batch_1` targets had finished without the historical + exception, but `transferred.checkpoint.h5` was still absent and final stacked + validation had not observably returned. Writers and the guard remained live; + terminal pool, manifest, gates, logbook, and exit artifacts remained absent. +- Updated and committed the required `FINAL_REPORT.md` at `dc4055c5` with the + current diagnosis, exact assigned surface, regression evidence, fresh + 530-test/static verdict, and corrected nonterminal host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or +mutate the release chain as a side effect of that external verification. + +## Owner-reported host failure continuation — 2026-08-21 07:59Z + +### State + +In progress. The supplied traceback is now proven to come from historical +runtime `33bf52fe`, not the current executable: its exact line 4512 validates +every target and line 4310 raises the reported strict family-binding error. +Complete as of 2026-08-21 08:34:04Z. The current assigned-only runtime +correction is intact, independent call-path and test audits found no remaining +canonical leak, and the strengthened ordinary/banked regression now sends the +real generated taxable-interest receipt through the canonical terminal +validator. The regression is proven red on the historical runtime and green on +the current runtime; the focused matrix, all 530 directly affected tests, the +complete changed 259-test file, and static/object checks are green. The +required final report is committed at `17367506`. The active restricted host +retry remains an external, nonterminal certification boundary. + +### Done + +- Re-read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the GitNexus query/context tools are unavailable in this session; + direct raise-site, caller, Git-object, host-log, and focused-test tracing is + the documented fallback. +- Preserved the owner-provided untracked failure and memory-guard artifacts. +- Reopened the committed root journal before making executable or test edits. +- Bound the traceback byte-for-byte to `33bf52fe`. At that revision the real + 15-target `puf_tax_itemization` family was split into physical + `puf_tax_itemization__batch_1`; the receipt builder attached QRF evidence to + every imputed target, and the validator compared unassigned taxable interest + with the public family before consulting calibration ownership. +- Traced the current early and late production paths. Both stacked owners + derive `regime_evidence_targets` from the immutable nine-target calibration + registry; ordinary and banked transfer retain regimes only for that + selection; receipt builders use the same selection; and validators reject + undeclared evidence before strict binding. All generic callers retain the + empty default. No alternate canonical producer, serializer, aggregate, or + calibration-write path broadens the selection. +- Confirmed exact binding remains fail-closed: the current validator does not + legalize `__batch_N` aliases. The historical defect was evidence-surface + leakage, not a reason to weaken record binding and not a value-calibration + leak. +- Confirmed the existing exact regression is genuinely failing-first: commit + `22b2c6bc` contains it while both relevant runtime objects are identical to + `33bf52fe`; the assigned-only runtime correction begins at `176c60fc` and is + hardened through `94b7aecb`. +- Ran the exact validator regression and the real ordinary and banked + 15-target producer regressions under the 12 GiB guard. All 3 passed with + exit zero and 0.569 GiB maximum observed per-process RSS. +- Reconciled the regression audit. The exact canonical validator fixture and + real ordinary/banked producer are both covered, but the real generated + taxable-interest receipt is not currently passed into the canonical + validator because test-authority execution deliberately skips production + validation. +- Ran all 530 directly affected tests across ordinary ACS transfer, multispine + serialization, stacked spine, multispine pool tooling, and H5 receipt I/O + together under the owner guard. All passed with exit zero and 1.534 GiB + maximum observed per-process RSS. Output was limited to the known joblib + physical-core fallback and pandas fixture-fragmentation warnings. +- Strengthened the real 15-target ordinary/banked regression by grafting its + generated taxable-interest receipt into a canonical gap-fill receipt and + invoking `validate_stacked_gap_fill_receipt`. This closes the only identified + target-receipt-to-terminal-validator coverage gap without changing executable + behavior or weakening exact record-family binding. +- Ran the strengthened ordinary and banked cases plus the exact forged-evidence + validator regression under the guard. All 3 passed with exit zero and 0.585 + GiB maximum observed per-process RSS; Ruff lint and formatting checks on the + edited test file also pass. +- Proved the strengthened regression failing-first in an isolated temporary + worktree. With only `acs_multispine.py`, `acs_transfer.py`, and + `stacked_spine.py` replaced by their exact `33bf52fe` objects, both ordinary + and banked cases failed at historical lines 4512 and 4310 with the supplied + taxable-interest record-binding error. Restoring the current three objects + made the identical two cases pass. The temporary worktree and registration + were removed, and the shared worktree remained unchanged. +- Received an independent approval of `ad2a44c1`. The review confirmed the + bridge exercises both fit implementations, cannot pass vacuously because + selected unemployment must retain evidence, and is appropriately scoped to + the reported target. No further executable or test change was recommended. +- Ran the complete changed `test_us_stacked_spine.py`: all 259 tests passed + with exit zero and 1.021 GiB maximum observed per-process RSS. Output was + limited to the known joblib fallback and pandas fixture-fragmentation + warnings. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range whitespace checks, and worktree + whitespace checks; all passed. +- Compared the current tree with complete-suite checkpoint `d29a8705`. + `packages/microcosm-build/src`, `tools`, `specs`, `pyproject.toml`, and + `uv.lock` are exact Git-object matches. The only package difference is the + strengthened stacked-spine test file, whose full 259-test suite is green. +- Corrected the earlier host launch inference. The active retry began at + launch-window commit `3194df71`, not `8920193e`; the production source tree, + build tool, and `stacked_spine.py` objects are identical at both commits and + at current `HEAD`. The original failed run began under journal-only + `f7ecac75`, whose complete build source and stacked-spine object match + `33bf52fe` exactly. Neither run embeds a Microcosm revision in its artifacts. +- Took a read-only host snapshot at `2026-08-21 08:28:13Z`. PID 28857 remained + active, the retry had 60 successful and zero failed primary-QRF stages, and + it had already passed the old taxable-interest boundary. Final `pool.h5`, + manifest, gates, and a terminal exit marker were still absent, so no host + certification is claimed. +- Refreshed and committed `FINAL_REPORT.md` at `17367506` with the exact root + cause, assigned-only correction, strengthened ordinary/banked red-green + evidence, affected and changed-file test results, static/object checks, + corrected host revision forensics, and precise non-certification boundary. +- Received an independent final fact-check approval of the report and current + journal state with no concrete factual issue. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or +mutate the release chain as a side effect of that external verification. + +## Reopened owner verification — 2026-08-21 + +### State + +Complete as of 2026-08-21 07:58:40Z. The supplied line-4512-to-line-4310 +traceback is bound exactly to historical runtime `33bf52fe`, while the current +tree contains the assigned-only correction. The focused validator and real +ordinary/banked wide-family regressions, all 530 directly affected tests, +static checks, and exact executable-tree checks are green. Three independent +audits found no remaining canonical production leak. The required final report +is committed at `17b3c06d`. Restricted host certification remains external and +nonterminal. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the GitNexus query/context tools are unavailable in this session; + direct raise-site, caller, history, and focused-test tracing is the documented + fallback. +- Confirmed `PROGRESS.md` and the required `FINAL_REPORT.md` already exist and + preserved the owner-provided untracked failure and memory-guard artifacts. +- Inspected the prior completion ledger without assuming its verdict remains + current for this continuation. +- Matched the supplied traceback exactly at `33bf52fe`: the old early validator + called `_validate_acs_imputed_pattern_evidence` for every target before + consulting the early calibration registry, so the physical + `puf_tax_itemization__batch_1` record for unassigned taxable interest was + compared with the canonical public family and failed record binding. +- Confirmed the current early and late stacked producers pass an explicit + registry-derived `regime_evidence_targets` selection to ordinary or banked + ACS transfer; the immutable registry contains two early and seven late + targets and excludes taxable interest. +- Confirmed the current early and late receipt builders attach QRF evidence + only to that same selection, while terminal validators reject any undeclared + evidence before invoking strict QRF record binding for a selected target. +- Ran the exact taxable-interest validator regression and the real ordinary and + banked 15-target transfer regressions under the owner 12 GiB guard: all 3 + passed, with 0.572 GiB maximum observed per-process RSS. Taxable interest + retained physical family `puf_tax_itemization__batch_1`, empty regimes, and + no QRF receipt evidence in both execution modes. +- Reconciled independent runtime, regression, and host/history audits. They + traced all four production `transfer_acs_inputs` caller classes and found no + alternate producer, ordinary/banked fit, serializer, receipt, validator, or + calibration-write route that broadens the registry-derived selection. +- Confirmed the exact validator regression is genuinely failing-first: commit + `22b2c6bc` contains the test while its runtime is byte-identical to failing + `33bf52fe`, so the old runtime reaches the supplied lines 4512 and 4310 rather + than the corrected undeclared-evidence boundary. +- Recorded the low-level API qualification: an explicit noncanonical library + caller may request regime provenance for any target on its requested surface. + No production caller exposes this choice; generic production callers use the + empty default, and canonical stacked validators reject unassigned evidence. +- Read the reused host directory without modifying it. The original failed log + and checkpoints were overwritten by the retry script, so the traceback now + survives only in the owner artifact and committed journals. The active retry + began under launch-window journal commit `8920193e`, completed the 47/47 + survey, 1/1 housing, and 10/10 primary-QRF checkpoints without the old error, + but remained active and lacked final pool, manifest, gates, exit marker, and + a Microcosm revision binding. It is not a certification verdict. +- Ran all 530 tests across ordinary ACS transfer, multispine serialization, + stacked spine, multispine pool tooling, and H5 receipt I/O together under the + owner guard. All passed with exit zero and 1.658 GiB maximum observed + per-process RSS. Output was limited to the known joblib physical-core fallback + plus 2,313 pandas fixture-fragmentation warnings. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range whitespace checks, and worktree + whitespace checks; all passed. +- Compared the current executable tree with complete-suite checkpoint + `d29a8705`. Every production source, tool, spec, project, and lock object is + identical. The sole package difference is the ordinary/banked parametrization + in `test_us_stacked_spine.py`, whose entire 259-test file passed inside the + affected run. At `d29a8705`, guarded fresh-process shards covered all 6,608 + collected repository items without a failed shard. +- Took the final read-only host snapshot at `2026-08-21 07:57:18Z`. The retry + remained active after passing the old failure point and completing its early + and primary-QRF checkpoints, but still had no final pool, manifest, gates, + exit marker, or Microcosm revision binding. No certification is claimed. +- Refreshed and committed `FINAL_REPORT.md` at `17b3c06d` with the root cause, + correction lineage, assigned-only reachability audit, failing-first + regression evidence, focused and affected-suite results, static/tree checks, + and precise host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted 1% host result with +passing `pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or +mutate the release chain as a side effect of that external verification. + +## Active owner continuation — 2026-08-21 07:11Z + +### State + +Complete as of 2026-08-21 07:34:35Z. The historical failure is diagnosed, the +assigned-only implementation is intact, the exact binding regression now +covers ordinary and banked wide-family execution, all 530 affected tests and +static checks are green, and `FINAL_REPORT.md` is committed at `96be5daf`. +Restricted host certification remains external and has no terminal verdict. +The supplied traceback fingerprints historical commit `33bf52fe`. The current +implementation contains the assigned-only repair and the exact failing-first +regression passes. Independent call-path and test audits found no remaining +canonical behavior leak, but identified one useful coverage asymmetry: the real +15-target taxable-interest transfer was exercised only with a target bank. The +ordinary-path parity case is now green; the affected suite remains to rerun. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the GitNexus MCP query/context tools are unavailable while the + local CLI is installed; local graph/CLI and direct source tracing are the + prescribed fallback. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Confirmed `FINAL_REPORT.md` is the existing required output file. +- Inspected the committed historical journal and correction ledger without + treating their prior completion claim as current verification. +- Generated a current GitNexus graph as required by the debugging workflow. + Analysis completed through a 563-file, 11,597-node, 31,245-edge graph, but + its final registry write was sandbox-blocked at + `~/.gitnexus/registry.json`, so query/context could not consume it. Removed + the generated 100 MiB local cache and used direct source/history tracing. +- Bound the supplied traceback exactly to `33bf52fe`: line 4512 validated QRF + evidence for every target before looking up the target in the early + calibration registry, and line 4310 required the canonical unsplit family. + The actual taxable-interest provenance used the bounded physical family + `puf_tax_itemization__batch_1`, causing the reported mismatch. +- Confirmed the historical transfer runtime also detected, verified, and + attached regimes globally. The value-calibration write path was already + registry-filtered; the leak was shared diagnostic/receipt behavior. +- Enumerated the live immutable registry: exactly two early and seven late + targets across the assigned model-required, source-operator, and adult-care + families. `taxable_interest_income` is absent. +- Traced every current production caller. Only the early and late stacked + owners pass `regime_evidence_targets`, both derive it from that registry, + ordinary and banked fits compute/verify regimes only for the selection, + unassigned target records have empty regimes, and receipt builders attach + evidence only for the same selection. Generic multispine and pool-tool + callers retain the empty default. +- Confirmed both terminal validators validate the four legacy transfer counts + for every target, then reject undeclared QRF/calibration evidence before + invoking strict QRF record binding only for a declared calibration target. +- Ran the exact taxable-interest validator regression and the real banked + 15-target integration regression on current `7668825b`: 2 passed, with + 0.573 GiB maximum observed per-process RSS. +- Ran the exact regression against detached test-only commit `22b2c6bc`, whose + runtime object is the failing `33bf52fe` version. It failed at the supplied + `stacked_spine.py:4512` caller and `:4310` raise with the same taxable-interest + record-binding error, at 0.454 GiB peak RSS. +- Reconciled independent call-path, regression-surface, and read-only host + audits. None found a current alternate canonical producer, fit, serializer, + validator, or calibration-write path that broadens the nine-target policy. +- Read the live host retry without changing it. At 2026-08-21 07:16:16Z it had + rebuilt taxable interest as target 22/47 and continued through target 39/47 + without the old traceback. It remained active, had no terminal exit or final + pool/manifest/gates artifacts, and recorded no Git revision, so this is not a + certification verdict. +- Parameterized the real 15-target `puf_tax_itemization` integration regression + over ordinary and banked execution. Both paths now prove the physical + taxable-interest record remains `puf_tax_itemization__batch_1` with empty + regimes and no QRF receipt, while selected unemployment retains regimes and + evidence. +- Ran the strengthened three-case focused matrix (exact validator plus ordinary + and banked real transfers): 3 passed, with 0.584 GiB maximum observed + per-process RSS. +- Ran all 530 tests across ordinary ACS transfer, multispine serialization, + stacked spine, multispine pool tooling, and H5 receipt I/O together under the + owner guard. All passed with exit zero and 1.643 GiB maximum observed + per-process RSS. The only output beyond progress was the known joblib + physical-core fallback plus 2,313 pandas fixture-fragmentation warnings. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range whitespace checks, and worktree + whitespace checks; all pass. +- Compared the current tree with complete-suite checkpoint `d29a8705`. The only + differences are the root journals and the one stacked-spine test file + strengthened here; every production source/tool/spec/project/lock object is + unchanged. At `d29a8705`, guarded fresh-process shards covered all 6,608 + collected repository items without a failed shard, and the complete changed + 259-test stacked-spine file is green now. +- Took a final read-only host snapshot at 2026-08-21 07:32:33Z. The retry had + written all 47 survey-target checkpoints and the 1/1 housing checkpoint, + including taxable interest, without the old traceback. The top-level host + directory still had only `build.log` and `guard.log`: no terminal exit, + `pool.h5`, manifest, gates, or Git-revision binding. This remains progress, + not certification. +- Refreshed and committed the required `FINAL_REPORT.md` at `96be5daf` with the + current root cause, correction lineage, ordinary/banked regression evidence, + guarded 530-test verdict, static checks, complete-suite object binding, + continuation commits, and precise host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted host result with passing +`pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or mutate +the release chain as a side effect of that external verification. + +## Active owner continuation — 2026-08-21 06:49Z + +### State + +Complete as of 2026-08-21 07:09:56Z. The supplied line numbers resolve to +historical commit `33bf52fe`, while the present branch contains the complete +assigned-only correction and exact failing-first regression. Three independent +audits, 14 decisive cases, all 529 directly affected tests, static checks, and +exact complete-suite object binding found no current gap that warrants +duplicate executable or test edits. `FINAL_REPORT.md` is committed with the +current evidence. Restricted host certification remains external and +incomplete. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed the GitNexus query/context tools are unavailable in this session; + direct raise-site, caller, producer, serializer, validator, test, and Git + history tracing will be used as the documented fallback. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Confirmed `FINAL_REPORT.md` is the existing required output file. +- Reopened and committed this progress journal before implementation work. +- Matched the supplied line 4512 caller and line 4310 raise exactly at + `33bf52fe`: the old path validated every target against its public family + before consulting the early calibration registry. +- Confirmed the canonical 15-target `puf_tax_itemization` family is bounded at + eight targets, so unassigned taxable interest carried the physical + `puf_tax_itemization__batch_1` record that triggered the reported mismatch. +- Enumerated the immutable policy registry: two early and seven late targets, + all within the assigned model-required, source-operator, and adult-care + surface; taxable interest is absent. +- Traced the current correction lineage. `22b2c6bc` added the exact boundary + regression; `176c60fc` made regime work/evidence default-empty and explicitly + registry-selected; `0b4339d1`, `887df056`, `94b7aecb`, and `21a48ba5` + hardened legacy counts, banked/wide-family behavior, canonical width, and + exact selected-target record binding. +- Confirmed both current stacked owners derive `regime_evidence_targets` only + from the registry; unselected records receive empty regimes, receipt builders + omit their evidence, and both validators reject undeclared evidence before + invoking strict assigned-target record binding. +- Read the mutable host log without changing it. A later retry has rebuilt + taxable interest successfully as target 22/47 and continued through 23/47, + but has no terminal exit, final pool, manifest, gates, or explicit Microcosm + revision binding, so it is not a certification verdict. +- Ran the exact taxable-interest boundary regression, the real banked + 15-target producer regression, and all 12 fully rehashed QRF-structure + mutations under the owner-provided 12 GiB/20 ms guard. All 14 passed in + 15.47 seconds with 0.568 GiB maximum observed per-process RSS. +- Demonstrated the exact regression is failing-first in a detached temporary + worktree at `22b2c6bc`, where the regression exists but the runtime still + matches `33bf52fe`. It followed lines 4512 and 4310 and failed with the same + taxable-interest record-binding error at 0.439 GiB peak RSS; the temporary + worktree was then removed. +- Reconciled three independent read-only history, runtime/reachability, and + regression/host audits. They found no alternate canonical selection, + producer, ordinary/banked fit, warm-bank, serializer, receipt, validator, or + calibration-write path that broadens the nine-target registry. No material + regression gap remains for the reported early gap-fill failure. +- Ran all 529 tests across ordinary ACS transfer, multispine serialization, + stacked spine, multispine pool tooling, and H5 receipt I/O together under the + owner guard. All passed in 12:12 with 1.643 GiB maximum observed per-process + RSS. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range whitespace checks, and worktree + whitespace checks; all passed. +- Compared exact Git objects for all `packages`, `tools`, `specs`, + `pyproject.toml`, and `uv.lock` against complete-suite checkpoint `d29a8705`; + every object matches. That committed checkpoint collected all 6,608 current + repository items and ran them in guarded fresh-process package/file shards + with no failed shard. +- Took a final read-only host snapshot at 2026-08-21 07:08:01Z. The mutable log + had rebuilt the exact taxable-interest target and continued through target + 34/47 without the old error, but no final pool, manifest, gates, terminal + exit, or explicit Microcosm revision binding exists. This remains progress, + not certification. +- Refreshed and committed the required `FINAL_REPORT.md` at `54dc7fbf` with + the current root cause, correction lineage, assigned-only reachability audit, + failing-first proof, guarded verification, complete-suite binding, commit + ledger, and host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted host result with passing +`pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or mutate +the release chain as a side effect of that external verification. + +## Reopened owner continuation — 2026-08-21 + +### State + +Complete as of 2026-08-21 06:46:04Z. The supplied traceback exactly +fingerprints the historical global-QRF leak, while current production +selection, evidence attachment, serialization, and validation are confined to +the immutable nine-target registry. The exact 14-case boundary matrix and all +529 directly affected tests pass under the owner guard, and static checks pass. +No duplicate executable or regression edit is warranted. `FINAL_REPORT.md` is +committed with the current evidence. The owner-provided host artifacts remain +untouched and untracked; the active host retry is not claimed certified. + +### Done + +- Replaced the stale required output with the current diagnosis, correction + lineage, regression evidence, guarded verification, static verdict, and + timestamped host boundary; committed it as `7180f6ef`. +- Re-ran the GitNexus error query. The CLI could not resolve this worktree + because its global registry contains only unrelated repositories and the + sandbox cannot register a Microcosm index; completed the skill workflow by + direct raise-site, history, caller, producer, serializer, and validator + tracing. +- Independently confirmed the supplied line 4512-to-4310 traceback at + `33bf52fe`: the old validator bound every target's physical record family to + the canonical public family before checking assignment. The 15-target PUF + itemization family split at width eight, so taxable interest carried + `puf_tax_itemization__batch_1` and failed the global binding check. +- Confirmed the correction lineage is already committed: `22b2c6bc` added the + failing-first exact regression; `176c60fc` added explicit selected-target + regime work/evidence and assignment-first validation; `0b4339d1`, + `887df056`, `94b7aecb`, and `21a48ba5` restored strict counts/family binding + and covered real banked, mixed-family, and rehashed-forgery cases. +- Re-audited ordinary, banked, warm-resume, early, late, serializer, and generic + caller paths. Only the two stacked owners opt in, both from the exact two + early/seven late registry; unassigned records have empty regimes, emit no QRF + evidence, and retain the four legacy transfer counts. No indirect unassigned + draw or write path was found. +- Ran the exact taxable-interest validator regression, the real banked + 15-target producer, and all 12 rehashed QRF structure mutations using the + worktree virtual environment under the 12 GiB/20 ms guard: 14 passed, with a + 0.572 GiB maximum observed per-process RSS. An initial `uv run` attempt did + not launch because the default cache is outside the writable sandbox. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files in one + guarded process. Pytest reached 100% with exit zero and no failures; maximum + observed per-process RSS was 1.665 GiB. +- Ran repository-wide `ruff check .`, Ruff format checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks; + all pass. Current executable source, tools, specs, lock, and root project + configuration are Git-identical to scoped-fix checkpoint `94b7aecb`. +- Reconciled three independent read-only audits. They found no current canonical + leakage path or material missing regression; no duplicate runtime or test + edit is warranted. +- Reopened the committed journal before taking implementation action in this + continuation and recorded the current verification boundary. +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed `FINAL_REPORT.md` is the existing required output file. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` files untouched. +- Inspected the clean tracked worktree and prior continuation history. +- Confirmed the GitNexus query/context tools and graph resources are unavailable + in this session; used direct raise-site, caller, producer, validator, test, + runtime-enumeration, and Git-history tracing as the documented fallback. +- Matched the supplied lines 4310 and 4512 exactly to `33bf52fe`. That source + validates QRF evidence on every target before consulting the assigned + calibration registry, so the unassigned taxable-interest target reaches the + strict public-family predicate with a bounded `puf_tax_itemization__batch_1` + record and raises the reported error. +- Enumerated the live immutable registry: two early and seven late targets, + covering only declared model-required, source-operator, and adult-care rows. + `person/puf_tax_itemization/taxable_interest_income` is absent. +- Traced both owner calls into `transfer_acs_inputs`, the ordinary and banked + fitting paths, both receipt builders, and both terminal validators. Regime + evidence defaults empty; owners pass only registry-selected targets; + unassigned records carry empty regimes; builders omit their QRF evidence; + validators reject any forged unassigned evidence before strict assigned + record-family binding. Legacy four-count accounting remains required for all + targets. +- Confirmed the canonical 15-target PUF tax-itemization family is pinned to the + certified width eight and splits into `__batch_1` and `__batch_2`; taxable + interest is in `__batch_1` but is not selected for calibration or evidence. +- Confirmed current executable Git trees under `microcosm-build/src`, `tools`, + and `specs` exactly match the prior complete-suite checkpoint `a5be536f`. +- Demonstrated the exact regression is failing-first in a detached temporary + worktree at `22b2c6bc`, whose runtime is byte-identical to `33bf52fe`. It + failed through lines 4512 and 4310 with the supplied taxable-interest + record-binding error (0.452 GiB peak); the worktree was then removed. +- Ran the exact taxable-interest validator regression, the real banked + wide-family producer, and all 12 rehashed QRF structure mutations on current + `HEAD` under the owner 12 GiB/20 ms guard. All 14 passed with a 0.570 GiB + maximum observed per-process RSS; the only warning was joblib's logical-core + fallback. +- Collected the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files at 64, 5, + 258, 164, and 38 tests respectively: 529 total. +- Ran all 529 affected tests together under the owner guard with warning + summaries disabled. Pytest reached 100% with exit zero and no failures; + maximum observed per-process RSS was 1.589 GiB. +- Reconciled three independent read-only source, history, regression, warm-bank, + and host audits. All found the current canonical path assigned-only, no + alternate leakage path, and no material missing regression for the supplied + failure. Selection does not change unassigned draws, calibration writes only + selected target columns, and warm banks persist raw draws/state while regime + evidence is recomputed from the current selection. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and committed-range plus worktree whitespace + checks; all passed. The first format invocation passed its newline-separated + list as one zsh argument and exited before checking files; the null-delimited + rerun checked all 15 successfully and made no changes. +- Collected the current repository under the owner guard: exactly 6,608 items + across 260 files, with a 1.163 GiB maximum observed per-process RSS. +- Confirmed `packages`, `tools`, `specs`, `pyproject.toml`, and `uv.lock` are + byte-for-byte Git-identical to complete-suite checkpoint `d29a8705`. That + checkpoint's fresh-process shards cover the same exact 6,608 items without + failures; only `PROGRESS.md` and `FINAL_REPORT.md` differ on current `HEAD`. +- Took a read-only host snapshot at 2026-08-21 02:04:45 EDT. The current retry + PID was alive, `build.log` had been truncated at 02:00:52 and remained zero + bytes, and `guard.log` had a fresh resource-wait heartbeat. A prior retry + rolled over without a gates artifact. No terminal exit, checkpoint tree, + pool, manifest, gates, spool, or receipt artifact existed, and no artifact + binds a Microcosm Git SHA. The supplied binding traceback is absent from the + live log and survives only in the owner note; the host is not certified. +- Took the final read-only snapshot at 02:21:24 EDT. PID 28857 still held the + zero-byte `build.log` open; the latest guard heartbeat was 02:19 EDT; and no + checkpoint, pool, manifest, gates, spool, receipt, exit, or revision-binding + artifact existed. +- Rewrote and committed `FINAL_REPORT.md` at `69637168` with the root cause, + exact assigned surface, correction chain, failing-first proof, guarded + focused and affected results, complete-suite object binding, static checks, + continuation commits, and remaining host boundary. + +### Next + +Obtain a durable, terminal, revision-bound restricted host result with passing +`pool.h5`, `pool.manifest.json`, and `pool.gates.json`. Do not publish or mutate +the release chain as a side effect of that external verification. + +## Current continuation — 2026-08-21 + +### State + +Complete. The supplied traceback fingerprints historical commit `33bf52fe`, +while the current tree contains the narrow assigned-target correction and exact +taxable-interest regression. Current source, history, runtime enumeration, and +two independent audits find no remaining scope leak or reason for a duplicate +executable edit. The regression is demonstrably failing-first; all 14 decisive +cases, all 529 directly affected cases, and the exact 6,608-item repository +collection pass without failures in guarded fresh processes. Lint, formatting, +whitespace, and executable-drift checks pass. `FINAL_REPORT.md` contains the +final continuation report. The restricted host retry remains an external, +unterminated certification boundary and is not claimed green. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed GitNexus MCP graph query/context tools and resources are unavailable + in this session. The local CLI built a 563-file graph but could not register + it because its hard-coded global registry is outside the writable sandbox; + moved the generated cache to the recoverable + `/private/tmp/microcosm-pkg3-two-part-gitnexus-23236043` location and used + direct raise-site, caller, source, test, and Git-history tracing as the + prescribed fallback. +- Confirmed `FINAL_REPORT.md` is the existing charter output file. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Reopened this committed journal before implementation work. The branch + carries earlier scoped-fix and regression commits, but their claims remain + untrusted pending current-tree diagnosis and verification. +- Matched the supplied lines 4310 and 4512 exactly to `33bf52fe`. That revision + attached QRF evidence to every imputed target and validated every target + before consulting the calibration registry. The canonical 15-target + `puf_tax_itemization` family splits at width eight, so taxable interest + carried record family `puf_tax_itemization__batch_1` while validation expected + the unsplit public family and raised at the reported predicate. +- Enumerated the immutable live registry: exactly two early and seven late + model-required, source-operator, and adult-care targets. Taxable interest is + absent. +- Traced current ordinary and banked transfer fitting, both canonical stacked + owners, both receipt builders, and both terminal validators. Regime work + defaults off; owners select only immutable specs; unassigned records and + receipts omit regimes and QRF evidence; validators reject forged unassigned + evidence before record binding; all targets retain strict four-count transfer + accounting; assigned targets retain exact family binding. +- Confirmed correction history: `22b2c6bc` adds the exact failing-target + regression; `176c60fc` supplies the primary opt-in runtime correction; + `887df056` restores strict record-family binding and independent legacy + counts; `94b7aecb` closes mixed-family/count/width gaps; and `21a48ba5` + rejects a plausible rehashed assigned `__batch_1` alias. +- Confirmed the current regression surface includes both the exact synthetic + taxable-interest validator boundary and a real banked wide-family producer + that proves taxable interest has no regimes or evidence while assigned + unemployment compensation retains both. +- Reconciled two independent read-only invariant and branch-history audits; + both reached the same root cause and current scoped-fix verdict. +- Inspected the mutable host log. It has been overwritten by a later retry that + records taxable interest rebuilt successfully at survey target 22/47 without + the supplied traceback, but it carries no revision binding and is not treated + as terminal certification evidence. +- Demonstrated the exact regression failing-first in an isolated temporary + worktree at `22b2c6bc`, where the test exists but the runtime remains + unchanged from `33bf52fe`. It followed the supplied line 4512-to-4310 path + and emitted the same taxable-interest record-binding error. Peak observed + per-process RSS was 0.425 GiB; the temporary worktree was then removed. +- Ran the exact taxable-interest boundary, the real banked wide-family + producer, and all 12 rehashed QRF structure mutations on current `HEAD` + under the owner 12 GiB/20 ms guard. All 14 cases passed with 0.564 GiB peak + observed per-process RSS. The only warning was joblib's logical-core + fallback. +- Collected the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files at 64, 5, + 258, 164, and 38 tests respectively: 529 total. +- Ran all 529 affected tests together under the owner guard with warning + summaries disabled. Pytest reached 100% with exit zero and no failures; peak + observed per-process RSS was 1.658 GiB. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and committed-range plus worktree whitespace + checks; all passed. +- Compared exact Git objects for `microcosm-build/src`, `tools`, `specs`, + `pyproject.toml`, and `uv.lock` against all-build-suite checkpoint + `a5be536f`; every pair matched. Only committed regression hardening and + journals differ after that checkpoint. +- Ran the full repository collection in one guarded pytest process. It reached + 83% with no test failures, then the owner guard terminated the process at + 12.170 GiB observed RSS (exit 99). This is recorded as a resource-bound + non-verdict, not a green suite result; it indicates cumulative single-process + memory and will be replaced by fresh-process package/file shards. +- Collected 6,608 repository tests: 5,708 build, 201 calibrate, 276 data, 93 + fit, and 330 frame. +- Ran the four smaller packages in fresh guarded processes. Fit passed 93 tests + at 0.724 GiB peak; calibrate passed 201 at 0.436 GiB; data passed 275 with one + skip at 11.049 GiB; and frame passed 294 with 36 skips at 6.488 GiB. These + account for all 900 non-build cases. +- Ran the build core+UK shard in a fresh guarded process: 1,856 passed and 33 + skipped in 7:44, with 4.182 GiB peak observed per-process RSS. +- Ran the 81-file build US a-r shard in a fresh guarded process: 2,406 passed + and three skipped in 23:29, with 8.844 GiB peak observed per-process RSS. +- Ran the 36-file build US s-z shard in a fresh guarded process: 1,411 passed + and one skipped in 26:20, with 10.579 GiB peak observed per-process RSS. +- Reconciled the shard collections. Their union exactly equals all 5,708 + collected build items and, with the four sibling packages, all 6,608 + repository items. No shard failed. The core+UK outcome summary includes two + additional passing subtest outcomes beyond its 1,887 collected items, which + explains why naively adding its pass/skip summary yields two extra outcomes. +- Took a final 2026-08-21 01:54:15 EDT read-only host snapshot. The external + runner remained alive with no current error match, but no runner exit, + `pool.h5`, pool manifest, or gates artifact existed. Its artifacts do not + explicitly bind a Microcosm Git SHA, so no host certification is claimed. +- Re-ran repository lint, scoped formatting, worktree and committed-range + whitespace, and exact executable-tree checks after the complete suite; all + passed. The only untracked files are the preserved owner-provided failure and + memory-guard artifacts. +- Rewrote and committed `FINAL_REPORT.md` at `6f162239` with the current root + cause, correction, red/green proof, focused/affected/full-suite results, + static verification, current commits, and remaining host boundary. + +### Next + +Obtain a terminal, revision-bound restricted host verdict with the final pool, +manifest, and gates artifacts. Do not publish or mutate the pending release +chain as a side effect of that verification. + +## Current continuation — 2026-08-21 03:29Z + +### State + +Complete. The supplied line numbers fingerprint historical commit +`33bf52fe`; the current tree already contains the complete scoped correction +and failing-first regression coverage. Three independent audits and 14 guarded +focused cases find no remaining leak or material coverage gap. All 529 tests +in the five directly affected files also pass under the owner guard. A +duplicate executable edit is not warranted. Lint, format, whitespace, and +executable-drift checks pass, and `FINAL_REPORT.md` contains the final +continuation report. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed GitNexus query/context tools are unavailable in this session; + using direct raise-site, caller, source, test, and Git-history tracing as the + prescribed fallback. +- Confirmed `FINAL_REPORT.md` is the existing charter output file. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Reopened this committed journal. Earlier scoped-fix and regression commits + exist on the branch, but their completion claims remain untrusted pending a + fresh current-tree diagnosis and verification. +- Reconstructed the exact historical failure. The canonical 15-target + `puf_tax_itemization` family is bounded at eight targets, placing taxable + interest in `puf_tax_itemization__batch_1`. At `33bf52fe`, QRF evidence was + attached to every imputed record and validated against the unsplit public + family before assignment was checked, producing the supplied line + 4512-to-4310 traceback. +- Enumerated the live immutable registry: exactly two early and seven late + model-required, source-operator, and adult-care targets. Taxable interest is + absent. +- Traced ordinary and banked transfers, both canonical owners, both receipt + builders, and both validators. Regime work defaults off; owners select only + immutable specs; unassigned provenance and receipts omit regimes and QRF + evidence; validators reject forged unassigned evidence before retaining + exact record-family binding for assigned targets; all targets retain strict + four-count transfer accounting. +- Reconciled independent runtime, history, and regression audits. All found the + correction complete and no canonical alternate path, warm-bank path, or + serializer that broadens the assigned selection. +- Confirmed the correction chain: `22b2c6bc` added the exact unassigned batched + regression; `176c60fc` supplied the primary opt-in runtime fix; `887df056` + restored strict exact-family binding and independent legacy counts; + `94b7aecb` pinned the canonical width and closed mixed-family/count gaps; and + `21a48ba5` rejects a fully rehashed plausible assigned `__batch_1` alias. +- Proved the primary regression is failing-first. Against the unchanged + `33bf52fe` source it fails at the supplied lines with the same taxable-interest + record-binding error; it passes against first fix `176c60fc` and current + `HEAD`. +- Ran the exact synthetic target, the real banked 15-target producer, and all + 12 rehashed QRF structure mutations under the owner 12 GiB/20 ms guard. All + 14 cases passed; maximum observed per-process RSS was 0.567 GiB. The only + warning was joblib's logical-core fallback. +- Ran all 529 cases in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the same guard. Pytest reached 100% with exit zero; maximum observed + per-process RSS was 1.657 GiB. A separate collection pass confirmed file + counts of 64, 5, 258, 164, and 38. +- Confirmed the complete current `microcosm-build/src`, `tools`, and `specs` + Git trees exactly match all-suite checkpoint `a5be536f`; the only affected + test difference is the three-line strict assigned-family forgery case. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and exact executable-tree comparisons against `a5be536f`; all passed. +- Refreshed and committed `FINAL_REPORT.md` at `6062fc3c` with the current + diagnosis, demonstrated red/green regression, guarded 14-case and 529-test + results, static verification, continuation commits, and explicit remaining + host boundary. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +## Current continuation — 2026-08-20 + +### State + +Complete. The supplied traceback is reproduced in source history, the existing +regression is proven red on the bad runtime and green on the correction, three +independent audits find no remaining leak, and focused, affected-suite, lint, +format, whitespace, and runtime-drift checks all pass. `FINAL_REPORT.md` +contains the final continuation report. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed GitNexus query/context resources are unavailable in this session; + using direct raise-site, call-chain, source, test, and history tracing as the + prescribed fallback. +- Confirmed `FINAL_REPORT.md` is the existing charter output file. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Reopened the committed journal for the supplied continuation. The branch + carries earlier scoped-fix and regression commits, but their claims remain + untrusted until revalidated against the supplied host log and current tree. +- Matched traceback lines 4310 and 4512 exactly to commit `33bf52fe`. That + revision validated QRF evidence for every early target before consulting the + calibration registry; the current source checks assignment first and rejects + evidence on unassigned targets without entering record binding. +- Enumerated the current immutable selector at runtime: exactly two early and + seven late model-required, source-operator, and adult-care targets. + `person/puf_tax_itemization/taxable_interest_income` is not selected. +- Re-ran the exact synthetic taxable-interest receipt, the real banked + 15-target producer, and all 12 strict rehashed QRF structure mutations under + the owner 12 GiB/20 ms guard. All 14 cases passed; maximum observed + per-process RSS was 0.568 GiB. The only warning was joblib's logical-core + fallback. +- Reconciled independent host-log, runtime, and regression-history audits. + Each found the correction complete and no alternative canonical path that + broadens evidence beyond the immutable nine-target selection. +- Demonstrated the existing regression is genuinely failing-first rather than + retrospective-only: `test_gap_fill_qrf_binding_excludes_unassigned_batched_targets` + fails against the unchanged `33bf52fe` runtime with the exact supplied + lines and message, then passes at first fix `176c60fc` and current `HEAD`. + The real banked producer and strict plausible-`__batch_1` forgery tests close + the remaining producer and permissive-validator seams, so a duplicate test + or executable edit is not warranted. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. Pytest reached 100% with exit zero; maximum observed + per-process RSS was 1.531 GiB. +- Confirmed the cited host log was overwritten by a mutable retry. That retry + rebuilt the bounded taxable-interest target and all 47 survey plus one + housing target without the old traceback, but it records no Git SHA and has + no terminal exit, pool, manifest, or gates artifact; it is progress evidence, + not revision-bound certification. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and committed-range plus worktree whitespace + checks; all passed. +- Compared the exact current `microcosm-build/src` and `us_runtime` Git tree + objects with all-build-suite checkpoint `a5be536f`; both match. The current + `tools`, `specs`, project metadata, and lockfile also have no drift from that + checkpoint. Its only affected-test difference is the committed three-line + strict assigned-family `__batch_1` forgery case, included in the current + 529-test pass. +- Refreshed and committed `FINAL_REPORT.md` at `1462c09d` with the current + diagnosis, demonstrated failing-first regression, guarded 14-case and + 529-test results, static verification, continuation commits, and explicit + remaining host boundary. +- Reconstructed the old failure at the cited lines. The 15-target + `puf_tax_itemization` family was split at the certified width of eight, so + `taxable_interest_income` recorded family `puf_tax_itemization__batch_1`. + Commit `33bf52fe` attached QRF evidence to every imputed target and validated + exact record binding against the unsplit public family before consulting the + calibration registry. +- Enumerated the live immutable selection at runtime: exactly two early and + seven late model-required, source-operator, and adult-care targets. + `person/puf_tax_itemization/taxable_interest_income` is absent. +- Traced ordinary and banked transfer fitting, both canonical stacked owners, + both receipt builders, both validators, and the generic serializers. Generic + regime work defaults off; canonical owners select only immutable specs; + unassigned records and receipts carry no regimes or QRF evidence; validators + reject forged unassigned evidence before retaining exact record-family + binding for assigned targets; all targets retain the mandatory four-count + transfer invariant. +- Reconciled independent runtime, target-scope, and regression/history audits. + Each found the current correction complete and no alternative canonical + caller that broadens the selection. +- Confirmed correction history: `176c60fc`, `887df056`, and `94b7aecb` scope and + harden the runtime; `22b2c6bc` covers the exact taxable-interest boundary; + `21a48ba5` rejects a fully rehashed plausible assigned `__batch_1` alias. +- Ran the exact synthetic host target, real banked 15-target producer, and all + 12 rehashed QRF structure mutations under the owner 12 GiB/20 ms guard. All + 14 cases passed; maximum observed per-process RSS was 0.571 GiB. The only + warning was joblib's logical-core fallback. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. Pytest reached 100% with exit zero; maximum observed + per-process RSS was 1.673 GiB. A separate collection pass confirmed file + counts of 64, 5, 258, 164, and 38. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and executable-tree comparison against all-build-suite checkpoint + `a5be536f`; all passed. Runtime source, tools, specs, project metadata, and + the lockfile have no drift from that checkpoint. Its only affected-test + difference is the committed three-line strict assigned-family regression. +- Took a final 03:02Z read-only host snapshot. The externally owned mutable + retry had rebuilt all 47 survey targets and the one housing target; bounded + taxable interest completed at target 22/47 without the supplied traceback. + The latest guard row still reported a resource wait, and no runner exit, + final pool, manifest, or gates artifact existed. The mutable host files do + not bind a Microcosm revision and are not accepted as terminal certification + evidence. +- Refreshed and committed `FINAL_REPORT.md` at `f1b6232d` with the current + diagnosis, 14-case and 529-test guarded results, static verification, + continuation commits, and explicit remaining host boundary. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +## Current continuation — 2026-08-21 02:14Z + +### State + +Complete. The supplied line numbers fingerprint historical commit `33bf52fe`, +while the current tree already contains the scoped runtime correction and exact +regression coverage. Independent source, history, regression, guarded test, +and static audits found no remaining leak and no reason for a duplicate +executable edit. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed GitNexus query/context tools are unavailable in this session and + selected direct raise-site, call-chain, source, test, and history tracing as + the fallback. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Confirmed `FINAL_REPORT.md` is the existing charter output file. +- Reconstructed the failure at `33bf52fe`: the validator called strict QRF + record binding for every target before consulting the calibration registry. + The real 15-target `puf_tax_itemization` transfer placed unassigned taxable + interest in `puf_tax_itemization__batch_1`, while validation expected the + unsplit public family and raised at the supplied line 4310. +- Confirmed taxable interest is absent from the immutable nine-target registry, + which contains exactly two early and seven late model-required, + source-operator, and adult-care targets. +- Traced the current correction through both ordinary and banked transfers, + both canonical owners, both receipt builders, and both validators. Regime + work defaults off; canonical callers select only immutable specs; unassigned + records and receipts carry no QRF evidence; validators reject forged + unassigned evidence before retaining exact record-family binding for assigned + targets; and all targets retain the mandatory transfer-count block. +- Audited the committed regression boundary: the exact synthetic taxable- + interest receipt, the real banked wide-family producer, and the fully rehashed + assigned `__batch_1` forgery jointly cover the failure and prohibit a + permissive family-alias workaround. +- Reconciled three independent invariant, history, and regression-seam audits; + all found the correction complete and no remaining executable or coverage + gap. +- Ran the exact synthetic taxable-interest boundary, the real banked 15-target + producer, and all 12 rehashed QRF structure mutations under the owner 12 GiB + guard. All 14 cases passed; maximum observed per-process RSS was 0.571 GiB. + The only warning was joblib's logical-core fallback. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and exact executable-tree comparisons against all-suite checkpoint + `a5be536f`; all passed. The current `microcosm-build/src`, `tools`, `specs`, + project, and lockfile content is unchanged from that checkpoint. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. The run reached 100% with exit zero and no failures; + maximum observed per-process RSS was 1.206 GiB. A separate collection pass + confirmed the 529-case bound. +- Took a final read-only host snapshot at 02:39Z. The mutable retry had rebuilt + survey target 26/47; bounded taxable interest completed at target 22/47 + without the supplied failure. The guard still reported a resource wait, and + no runner exit, final pool, manifest, or gates artifact existed. The mutable + files do not bind a Microcosm revision and are not accepted as terminal + certification evidence. +- Refreshed `FINAL_REPORT.md` with the current diagnosis, audit verdicts, + guarded 14-case and 529-test results, static verification, current commits, + and explicit remaining host boundary. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +## Current continuation — 2026-08-21 01:39Z + +### State + +Complete. The supplied traceback exactly fingerprints bad commit `33bf52fe`, +while current source already contains the complete scoped correction and +regression coverage. Fresh source, history, regression, guarded test, and +static audits found no remaining leak and no reason for a duplicate executable +edit. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed GitNexus query/context tools are not exposed in this session, so + the fallback is direct raise-site, caller, producer, validator, test, and + Git-history tracing. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Identified `FINAL_REPORT.md` as the existing charter output file. +- Reconstructed the historical failure at the cited lines: validation called + `_validate_acs_imputed_pattern_evidence` for every target before consulting + the calibration registry. The real 15-target `puf_tax_itemization` transfer + recorded `taxable_interest_income` under its bounded `__batch_1` family, + while validation expected the unsplit public family. +- Enumerated the immutable live selection: two early and seven late targets, + comprising one model-required, seven source-operator, and one adult-care + target. `taxable_interest_income` is absent. +- Independently traced both canonical producers, both receipt builders, and + both validators. Canonical callers derive the opt-in only from the immutable + specs; unassigned records carry empty regimes; unassigned receipts omit QRF + evidence; validators reject forged unassigned evidence before retaining + strict exact-family binding for assigned targets; and all targets retain the + mandatory four-count transfer block. +- Audited correction history. `22b2c6bc` added the failing-first exact-target + regression; `176c60fc`, `887df056`, and `94b7aecb` supplied and hardened the + runtime correction; `21a48ba5` protects exact binding from a fully rehashed, + plausible `__batch_1` alias. No source commit follows `94b7aecb`. +- Confirmed the current `microcosm-build/src`, `tools`, and `specs` trees are + unchanged from the all-build-suite checkpoint `a5be536f`; only the strict + assigned-family forgery regression differs in the affected test tree. +- Reconciled three independent read-only audits. Runtime, history, and + regression reviewers all found the correction complete and no material + coverage gap. Their focused selections passed 4 and 24 cases respectively. +- Ran the exact synthetic taxable-interest boundary, the real banked 15-target + producer, and all 12 fully rehashed QRF structure mutations under the owner + 12 GiB/20 ms guard. All 14 cases passed; maximum observed per-process RSS was + 0.572 GiB. The only warning was joblib's logical-core fallback. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. The run reached 100% with exit zero and no failures; + maximum observed per-process RSS was 1.858 GiB. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and executable-tree comparisons against `a5be536f`; all passed. The current + `microcosm-build/src`, `tools`, `specs`, project, and lockfile content is + unchanged from that all-suite checkpoint. Its only affected-test difference + is the committed three-line in-range-family forgery regression. +- Took a final read-only host snapshot at 02:10Z. The mutable retry had shown + progress into late `puf_tax_itemization__batch_3` at 01:45Z, but its + `build.log` was later truncated to zero bytes; the guard still reported a + resource wait, and no runner exit, final pool, manifest, or gates artifact + existed. The run does not bind a Microcosm revision, so it is not accepted as + terminal certification evidence. +- Refreshed `FINAL_REPORT.md` with the current diagnosis, independent audit + verdicts, guarded 14-case and 529-test results, static verification, current + continuation commits, and explicit remaining host boundary. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +## Current continuation — 2026-08-21 01:11Z + +### State + +Complete. The supplied traceback +fingerprints commit `33bf52fe`: it attached QRF regime evidence to every +imputed target and validated exact record binding before checking calibration +assignment. The current tree already contains the complete scoped correction +and exact regression coverage, so no duplicate runtime or test edit is +warranted. + +### Done + +- Read `CLAUDE.md` and the complete GitNexus debugging skill instructions. +- Confirmed that GitNexus query/context tools are not exposed in this session, + so direct raise-site, caller, producer, validator, test, and history tracing + will be used. +- Preserved the owner-provided untracked `_BUILD-FAILURE-1PCT.txt` and + `.codex-memory-guard.py` artifacts untouched. +- Confirmed `FINAL_REPORT.md` is the existing output file for this charter. +- Reconstructed the historical failure at the cited lines. The real 15-target + `puf_tax_itemization` family was split at width eight, so + `taxable_interest_income` carried record family + `puf_tax_itemization__batch_1`; the old validator compared it with the + unsplit public family before discovering that the target had no calibration + spec. +- Enumerated the live immutable selection: exactly two early and seven late + specs across the assigned model-required, source-operator, and adult-care + surfaces. `taxable_interest_income` is absent. +- Traced both canonical owners, transfer implementations, receipt builders, + and validators. Generic regime work defaults off; canonical callers opt in + only the immutable selected targets; unassigned records carry empty regimes; + unassigned receipts omit QRF evidence; and validators reject forged + unassigned evidence before retaining strict family binding for assigned + targets. The four legacy transfer counts remain mandatory for every target. +- Audited the correction history: `22b2c6bc` added the exact failing-target + regression; `176c60fc`, `887df056`, and `94b7aecb` form the complete runtime + correction; and `21a48ba5` protects strict binding from a fully rehashed, + plausible `__batch_1` alias. No source commit follows `94b7aecb`. +- Reconciled independent runtime and history audits. Both found no remaining + canonical behavior or evidence leak; read-only focused selections of three + runtime cases and seventeen history-selected cases passed. +- Confirmed the current `microcosm-build/src`, `tools`, and `specs` Git trees + exactly match all-build-suite checkpoint `a5be536f`. Only the committed + three-line strict-binding regression differs in the affected test tree. +- Ran the exact synthetic taxable-interest boundary, real banked 15-target + producer, and all 12 fully rehashed QRF structure mutations under the owner + 12 GiB/20 ms guard. All 14 cases passed; maximum observed per-process RSS was + 0.560 GiB. The only warning was joblib's logical-core fallback. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. The run reached 100% with exit zero and no failures; + maximum observed per-process RSS was 1.688 GiB. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and executable-tree comparisons against `a5be536f`; all passed. The current + `microcosm-build/src`, `tools`, and `specs` tree objects exactly match that + all-build-suite checkpoint. +- Refreshed `FINAL_REPORT.md` with the current diagnosis, three independent + audit verdicts, guarded 14-case and 529-test results, static verification, + continuation commits, and the explicit non-claim of a terminal host + certification result. +- Performed a final read-only host check at 01:37Z. The mutable retry had + completed the exact bounded taxable-interest target, all early survey and + housing targets, and entered the late-producer DAG without the supplied + traceback, but it still had no runner exit, final pool, manifest, gates + artifact, or Microcosm revision binding. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +The completion snapshots below are historical. + +## Current continuation — 2026-08-21 00:22Z + +### State + +Complete. The traceback line numbers exactly +identify bad commit `33bf52fe`, where every imputed target carried QRF regime +evidence and the canonical validator performed exact record binding before +checking calibration assignment. The current branch already contains the +complete scoped correction and exact regression coverage, so no duplicate +runtime or test edit is warranted. + +### Done + +- Read `CLAUDE.md` and the GitNexus debugging skill instructions. +- Preserved the owner-provided untracked failure and memory-guard artifacts. +- Confirmed that GitNexus query/context tools are not exposed in this session, + so direct source, call-site, regression, and Git-history tracing will be used. +- Reconstructed the failure: the real 15-target `puf_tax_itemization` transfer + emitted family `puf_tax_itemization__batch_1` for unassigned + `taxable_interest_income`, while the old validator expected the unsplit + public family before consulting the immutable calibration registry. +- Confirmed the current correction at `176c60fc`, `887df056`, and `94b7aecb`: + generic regime work defaults off; both stacked producers derive their opt-in + solely from the immutable calibration specs; both receipt builders omit QRF + evidence for unassigned targets; and both validators reject forged + unassigned evidence before retaining exact binding for assigned targets. +- Enumerated the live canonical selection: exactly two early and seven late + specs across model-required, source-operator, and adult-care families; + `taxable_interest_income` is absent. The 15-target public family remains on + the transfer surface but has no selected target. +- Audited the exact regressions already committed: the synthetic canonical + validator case rejects forged batched evidence and accepts the evidence-free + legacy receipt; the real banked wide-family producer proves the target's + record has no regimes or receipt evidence; and the assigned-target mutation + suite preserves strict record-family equality, including a rehashed + plausible `__batch_1` alias. +- Verified that the current `microcosm-build/src`, `tools`, and `specs` Git + trees exactly match all-build-suite checkpoint `a5be536f`; only the already + committed three-line strict-binding regression differs in the affected test + tree. +- Reconciled independent history and regression audits. Both found the scoped + correction complete with no later executable regression or material test + gap; the regression audit also ran its broader 17-case selection green. +- Reconciled the independent runtime audit. It traced both canonical producer, + receipt, and validator paths and found no remaining selector or binding leak; + its three-case focused runtime selection also passed. +- Ran the exact synthetic taxable-interest boundary, real banked 15-target + producer, and all 12 fully rehashed QRF structure mutations under the owner + 12 GiB/20 ms guard. All 14 cases passed; maximum observed per-process RSS was + 0.557 GiB. The only warning was joblib's logical-core fallback. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner guard. The run reached 100% with exit zero and no failures; + maximum observed per-process RSS was 1.283 GiB. A separate collection pass + confirmed file totals of 64, 5, 258, 164, and 38 tests. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, committed-range and worktree whitespace checks, + and an executable-tree comparison against `a5be536f`; all passed. No source, + tool, spec, project, or lockfile differs from that all-build-suite checkpoint. +- Refreshed `FINAL_REPORT.md` with the current diagnosis, three independent + audit verdicts, guarded 14-case and 529-test results, static verification, + continuation commits, and the explicit non-claim of a new host certification + run. + +### Next + +Run or identify a revision-bound restricted host 1% retry that reaches a +terminal runner verdict and emits the expected pool, manifest, and gates +artifacts; this remains external to the completed local correction. + +The completion snapshots below are historical. + +## State + +The 2026-08-21 00:20Z owner continuation is complete. Its independent +raise-site, producer, validator, history, and regression diagnosis found the +reported failure fingerprints `33bf52fe`: every imputed target carried QRF +regime evidence and validation performed exact record binding before checking +the nine-target calibration registry. The real 15-target transfer therefore +presented `puf_tax_itemization__batch_1` for unassigned +`taxable_interest_income` while validation expected the unsplit family. The +current correction is already complete in `176c60fc`, `887df056`, and +`94b7aecb`: generic evidence work defaults off; early and late owners select +the exact two and seven immutable specs; unassigned records and receipts carry +no QRF evidence; and validators reject forged unassigned evidence before +retaining exact binding for assigned targets. The exact synthetic failure, +real banked wide-family producer, and all 12 rehashed structure mutations pass +(14 cases, 0.569 GiB peak under the 12 GiB guard). No duplicate runtime or +test edit is warranted. The five-file affected suite is also green: all 529 +tests passed together with a 1.472 GiB peak under the same guard. Static +verification also passes: repository lint, formatting of all 15 correction- +range Python files, committed-range whitespace, and exact `src`, `us_runtime`, +`tools`, and `specs` tree comparisons with all-build-suite checkpoint +`a5be536f`. Only the strict three-line assigned-family forgery regression and +journals differ after that checkpoint. `FINAL_REPORT.md` contains the current +revision-bound report at `b523e70c`; no local implementation, regression, +verification, or reporting work remains. The owner-provided +`_BUILD-FAILURE-1PCT.txt` and `.codex-memory-guard.py` remain untracked and +untouched, and no terminal host result is claimed. + +The immediately preceding completion snapshot follows and is historical. + +The 2026-08-20 23:08Z owner continuation has independently reconfirmed the +raise-site diagnosis and found no remaining executable or regression gap. At +bad commit `33bf52fe`, every imputed target carried QRF regimes and terminal +validation performed exact record-family binding before consulting the +nine-target calibration registry. Current production derives the exact two +early and seven late selections from that immutable registry; both receipt +builders use the same selector; and both validators require the shared +four-count block, reject evidence on unassigned targets, then retain exact +binding for assigned targets. The exact synthetic taxable-interest case, real +banked 15-target producer, and all 12 rehashed structure mutations pass (14 +cases total) under the 12 GiB guard with 0.541 GiB maximum observed +per-process RSS. No duplicate executable edit is warranted. The affected-suite +rerun is also green: all 529 tests in the five directly affected files passed +under the same guard with 1.459 GiB maximum observed per-process RSS. Static +verification also passes: repository lint, formatting of all 15 correction- +range Python files, committed-range whitespace, and exact `microcosm-build/src` +and `us_runtime` tree comparisons with all-build-suite checkpoint `a5be536f`. +Only the strict three-line assigned-family forgery regression and journals +differ after that checkpoint. `FINAL_REPORT.md` now contains the current +revision-bound diagnosis, verification, and exact non-terminal host snapshot; +no local implementation, regression, verification, or reporting work remains. +The owner-provided +`_BUILD-FAILURE-1PCT.txt` and `.codex-memory-guard.py` remain untracked and +untouched, and no terminal host result is claimed. + +The immediately preceding completion snapshot follows and is historical. + +The 2026-08-20 23:05Z owner continuation is complete. It independently +confirmed the exact +failure path and current correction. At bad commit `33bf52fe`, receipt +production attached QRF evidence to every imputed target and terminal +validation checked exact record-family binding before consulting calibration +assignment. The host's real 15-target `puf_tax_itemization` transfer therefore +reported bounded family `puf_tax_itemization__batch_1` for unassigned +`taxable_interest_income`, while validation expected the public unsplit family. +Current runtime source is unchanged from the complete correction at +`94b7aecb`: generic regime work defaults off; canonical owners select exactly +the two early and seven late immutable specs; unselected records have no +regimes or receipt evidence; and validators reject evidence on unassigned +targets before applying strict binding to assigned targets. Independent +regression review found the exact synthetic failure, real banked wide-family +producer, and rehashed in-range family-forgery cases already committed and +adequate, so no duplicate executable edit is warranted. All 14 decisive +focused cases and all 529 tests in the five directly affected files pass under +the 12 GiB guard, with respective 0.568 GiB and 1.617 GiB maximum observed +per-process RSS. Repository lint, formatting of all 15 correction-range Python +files, committed-range whitespace, and exact source-tree comparisons with the +all-build-suite checkpoint also pass. `FINAL_REPORT.md` now contains the +revision-bound diagnosis, correction, regression, verification, and remaining +host boundary; no local implementation, regression, verification, or reporting +work remains. The owner-provided `_BUILD-FAILURE-1PCT.txt` and +`.codex-memory-guard.py` remain untracked and untouched; no terminal host +success is claimed. The final external check found `build.log` truncated to +zero bytes, continuing resource waits in `guard.log`, and no final output +artifacts or runner verdict. + +The opening checkpoint and earlier completion snapshots below are historical. + +The 2026-08-20 22:35Z owner continuation has completed an independent +raise-site, current-source, history, and regression audit of the supplied host +1% ACS QRF record-binding failure. Commit `33bf52fe` detected and copied QRF +regimes for every imputed target, receipt builders attached that evidence +globally, and terminal validation entered strict record binding before looking +up the nine-target calibration assignment. The real 15-target +`puf_tax_itemization` producer therefore supplied bounded family +`puf_tax_itemization__batch_1` for unassigned `taxable_interest_income` while +the validator expected the unsplit family. The complete current correction is +already committed in `176c60fc`, `887df056`, and `94b7aecb`: generic transfer +defaults evidence work off; canonical early/late producers and receipt builders +derive exact two-early/seven-late selections from the immutable specs; and both +validators reject evidence on an unassigned target before invoking exact +record binding only for assigned targets. Existing regressions cover the exact +synthetic host failure, the real banked 15-target producer, and the unsafe +permissive workaround via a fully rehashed in-range family alias. No material +runtime or test gap was found, so no duplicate executable edit is warranted. +The GitNexus workflow generated a local graph, but registration and subsequent +query/context calls were blocked by the sandboxed user-wide registry; its +112 MiB cache was removed. All 14 decisive focused cases and all 529 tests in +the five directly affected files now pass under the owner-provided 12 GiB/ +20 ms guard, with respective 0.550 GiB and 1.523 GiB maximum observed +per-process RSS. Repository lint, formatting of all 15 Python files changed +since the bad commit's parent, committed-range whitespace, and exact source-tree +comparisons with all-build-suite checkpoint `a5be536f` also pass. Final +reporting is complete in `FINAL_REPORT.md`; no local implementation, +regression, verification, or reporting work remains. The last read-only host +check found a still-growing 58,003-byte log after all early survey/housing and +several late-transfer checkpoints, but no runner exit, final artifacts, or +Microcosm revision binding, so no host success is claimed. The owner artifacts +remain untracked and untouched. + +The immediately preceding completion snapshot follows and is historical rather +than current state. + +The 2026-08-20 21:18Z owner continuation has completed independent source-flow, +history, and regression audits of the supplied host 1% ACS QRF record-binding +failure. The traceback exactly fingerprints bad commit `33bf52fe`: global QRF +evidence reached the real `puf_tax_itemization__batch_1` record for unassigned +`taxable_interest_income`, and terminal validation compared it with the +unsplit family before consulting the nine-target registry. The complete +current correction is the series `176c60fc`, `887df056`, and `94b7aecb`. +Generic transfer now defaults to no regime work; canonical early/late owners, +receipt builders, and validators independently select only the two early and +seven late immutable specs; and unassigned evidence is rejected before strict +record binding. The exact synthetic failure regression, real banked 15-target +producer regression, and fully rehashed in-range family-forgery regression are +already committed, so no duplicate runtime or test edit is warranted. Guarded +verification is green: all 14 decisive focused cases and all 529 tests in the +five directly affected files passed under the owner 12 GiB/20 ms guard, with +respective 0.562 GiB and 1.444 GiB maximum observed per-process RSS. Static and +tree-drift verification also passes: repository lint, formatting of all 15 +Python files changed since the bad commit's parent, committed-range whitespace, +and exact source-tree comparisons with all-build-suite checkpoint `a5be536f`. +Final reporting is complete in `FINAL_REPORT.md`; no local implementation, +regression, verification, or reporting work remains for the supplied failure. +No host success is claimed. The final read-only host check found a 51,315-byte +mutable log ending after survey 47/47 and housing 1/1, while the external guard +still reported a resource wait and no runner exit, pool, manifest, or gates +artifact existed. The owner-provided `_BUILD-FAILURE-1PCT.txt` and +`.codex-memory-guard.py` remain untracked and untouched. + +The immediately preceding completion snapshot follows and is historical rather +than current state. + +The current 2026-08-20 owner continuation has completed its independent +raise-site, source-flow, history, and regression diagnosis. The supplied line +numbers fingerprint bad commit `33bf52fe`: its globally enabled QRF evidence +reached the real `puf_tax_itemization__batch_1` record for unassigned +`taxable_interest_income`, then terminal validation compared that record with +the unsplit family before consulting the nine-target registry. The complete +current correction is the runtime series `176c60fc`, `887df056`, and +`94b7aecb`, not the first commit alone. Canonical producers now derive an exact +two-early/seven-late selection from the immutable specs, generic transfer +defaults to no regime work, receipts omit evidence for unassigned targets, and +validators reject forged evidence before calling strict record binding only on +assigned targets. The exact synthetic host regression, real 15-target banked +producer regression, and selected-family in-range alias regression are already +committed and jointly cover the failure and the unsafe permissive workaround; +no duplicate runtime or test edit is warranted. All 14 decisive focused cases +and all 529 tests in the five directly affected files pass under the owner +12 GiB/20 ms guard, with respective 0.568 GiB and 1.594 GiB maximum observed +per-process RSS. Repository lint, formatting of all 15 Python files changed +since the bad commit's parent, committed-range whitespace, and both runtime +tree-object comparisons with all-build-suite checkpoint `a5be536f` pass. Final +reporting is complete in `FINAL_REPORT.md`; no local implementation, regression, +or verification work remains for the supplied failure. No host success is +claimed. The owner-provided `_BUILD-FAILURE-1PCT.txt` and +`.codex-memory-guard.py` remain untracked and untouched. + +Earlier continuation snapshots follow and are historical rather than current +state. + +The 2026-08-20 20:22Z owner continuation has completed an independent +raise-site and history diagnosis against the current tree. Commit `33bf52fe` +computed QRF regimes and emitted evidence for every ACS target, and terminal +validation checked every target before consulting the nine-target calibration +registry. The wide `puf_tax_itemization` transfer therefore supplied the real +bounded record family `puf_tax_itemization__batch_1` while the validator +expected the unsplit family, causing the supplied failure on unassigned +`taxable_interest_income`. The current executable correction already present +from `176c60fc` makes regime work and receipt evidence explicit opt-ins derived +from the immutable assigned specs, checks assignment before QRF binding, and +rejects forged evidence on unassigned targets while retaining their mandatory +four-count transfer invariant. No executable file differs from the prior full +build-suite checkpoint `a5be536f`. The regression audit found that strict +selected-family binding lacked a plausible in-range `__batch_1` forgery, so +that case is now included alongside the existing forged and out-of-range +families. All 20 focused cases pass under the 12 GiB/20 ms guard with a 0.586 +GiB peak. All 529 tests in the five directly affected transfer, serialization, +stacked, pool-tool, and H5 files also pass in one guarded process with a 1.531 +GiB peak. Repository lint, formatting of all 15 Python files changed since the +bad commit's parent, committed-range whitespace, and runtime drift checks pass. +The runtime tree remains identical to `a5be536f`, where all 225 build-package +test files passed. Local correction, regression hardening, verification, and +reporting are complete. The last read-only host check found the mutable +`build.log` truncated to zero bytes, the external guard still waiting on other +work, and no pool, manifest, gates artifact, or terminal marker; no host success +is claimed. `FINAL_REPORT.md` contains the final revision-bound report. +The owner-provided `_BUILD-FAILURE-1PCT.txt` and `.codex-memory-guard.py` remain +untracked and untouched. + +The 2026-08-20 owner continuation independently revalidated the supplied host +1% traceback against the current executable tree. At `33bf52fe`, every ACS +target received regime provenance and terminal validation invoked the QRF +evidence validator before consulting the nine-target calibration registry. The +real wide-family record was `puf_tax_itemization__batch_1`, while the validator +expected unsplit `puf_tax_itemization`, causing the reported binding failure. +The current tree defaults regime work off, derives the canonical opt-in solely +from the nine immutable assigned specs, omits evidence from unassigned records +and receipts, and rejects forged evidence on those receipts before exact +selected-record binding. Two independent read-only audits found no remaining +canonical leak and no need for another executable edit. Unassigned generated +values and QRF behavior remain unchanged; the shared four-count receipt +invariant continues to apply to every canonical target. Verification is now in +progress: all eight focused scope and binding regressions pass under the 12 GiB +guard with a 0.559 GiB maximum observed per-process RSS, and all 528 tests in +the five directly affected files pass in one guarded process with a 1.494 GiB +peak. Repository lint, scoped formatting, and committed-range whitespace also +pass. Only the two root journals differ from `a5be536f`, so the executable tree +is identical to the checkpoint where all 225 `microcosm-build` test files +passed. Local diagnosis and verification are complete; the final report is +refreshed in `FINAL_REPORT.md`. No local implementation or verification work +remains. The owner-provided `_BUILD-FAILURE-1PCT.txt` and +`.codex-memory-guard.py` remain untracked and untouched. + +The 2026-08-20 owner continuation independently re-traced the reported host 1% +stacked gap-fill binding failure against the branch's existing scoped-QRF +correction. At `33bf52fe`, terminal validation called the QRF evidence validator +for every early transfer target before consulting the nine-target calibration +registry. The current tree consults that registry first, forbids evidence on +unassigned targets, and attaches or validates it only for the exact selected +targets. Seven focused tests pass, including both the synthetic failing boundary +and a real banked 15-target `puf_tax_itemization` transfer. All 528 tests in the +five affected transfer, stacked, serialization, pool, and H5 files pass under +the 12 GiB guard with a 1.531 GiB maximum observed per-process RSS. Repository +lint, touched-file formatting, committed-range whitespace, and the executable +tree comparison with the prior all-build-suite pass also succeed. No new host +artifact is claimed: the later host retry reaches every survey and housing +transfer checkpoint without this traceback but has no terminal marker or output +artifacts. The untracked owner artifacts `_BUILD-FAILURE-1PCT.txt` and +`.codex-memory-guard.py` are preserved unchanged. +The final continuation report is committed in `FINAL_REPORT.md` at `8328fd87`; +no local implementation or verification work remains for the supplied failure. + +The 2026-08-20 local continuation is complete for the host 1% binding failure +at `person/puf_tax_itemization/taxable_interest_income`. An independent audit +from the invariant raise site confirms the branch's existing scoped-binding +correction addresses the exact leaked surface without weakening record-family +binding. Six focused regressions and all 528 tests in the five directly +affected files pass on the current tree; repository lint, touched-file format, +and committed-range whitespace checks also pass. The executable tree is +unchanged from `a5be536f`, where all 225 `microcosm-build` test files passed. +An externally owned host 1% retry rebuilt all 47 survey targets and the one +housing target, including the exact bounded `taxable_interest_income` record, +without a traceback. Its build log then became quiet while the external guard +continued to report live build processes and no runner exit marker; terminal +stacked receipt validation is therefore not claimed. The supplied +`_BUILD-FAILURE-1PCT.txt` and `.codex-memory-guard.py` remain untracked owner +artifacts and will not be committed. + +The earlier local continuation concluded that the root cause was +global QRF regime detection and evidence on every ACS transfer target even +though the calibration owner declares only nine targets. Regime checks are +now default-off and must be explicitly selected by `(entity, target)`; the +stacked owner selects only the exact declared targets present on each +early/late surface. Unassigned records and receipts retain their legacy shape, +while selected evidence keeps exact record-family binding. All 225 +`microcosm-build` test files are green on the final tree across fresh pytest +processes, and repository lint, touched-file formatting, and committed-range +diff checks pass. Canonical production enforces the certified eight-target fit +width so selected receipt families remain exactly bound. No after artifact is +accepted: the active host-only 1% retry has no terminal verdict. The +uncontaminated 1% before artifact remains recorded at commit `5f5e5e91`; no +frozen battery band, threshold, comparator, seed, or fold has changed. + +The required `uv sync --all-packages --extra us` was attempted first. The +default cache is sandbox-read-only; a retry with a writable cache reached PyPI +but DNS is unavailable. Verification therefore uses the already-synced +`microcosm-707` environment, whose `uv.lock` SHA-1 matches this worktree, with +`PYTHONPATH` pinned to this worktree's five package source directories. + +## Done + +- Refreshed and committed `FINAL_REPORT.md` with this continuation's exact + diagnosis, focused and affected-suite results, static/tree verification, and + the final non-terminal host snapshot. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all pass. + Exact Git tree objects for all `microcosm-build/src`, its `us_runtime` + subtree, `tools`, and `specs` match checkpoint `a5be536f`. Since that full- + build-suite checkpoint, only the already-passed three-line binding + regression and journals changed. +- Re-ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files together + under the owner 12 GiB/20 ms guard. The run exited zero with a 1.472 GiB + maximum observed per-process RSS; a collection-only pass bound the current + file totals to 64, 5, 258, 164, and 38 respectively. +- Opened and committed this continuation journal before diagnostic work. +- Followed the GitNexus debugging workflow. The CLI generated a local graph, + but registration/query was blocked by the sandboxed user-wide registry; its + 112 MiB repo-local cache was removed with `gitnexus clean`. Direct raise-site, + caller, producer, validator, and history tracing established the full path. +- Independently confirmed the current canonical selector contains exactly two + early and seven late immutable specs and no `puf_tax_itemization` target. + Generic transfer defaults evidence work off, both receipt producers derive + the same selected set, and both validators branch on assignment before QRF + binding while retaining mandatory four-count validation for every target. +- Audited the existing regressions. Commit `22b2c6bc` already contains the + exact synthetic bounded-family failure, the real banked 15-target producer + proves the evidence never reaches `taxable_interest_income`, and + `21a48ba5` proves a fully rehashed plausible `__batch_1` alias remains + invalid for an assigned target. No duplicate regression is needed. +- Re-ran the exact failure, real producer, and all strict rehashed structure + mutations under the 12 GiB/20 ms guard: 14 passed with a 0.569 GiB maximum + observed per-process RSS. +- Refreshed `FINAL_REPORT.md` with this continuation's independent diagnosis, + current 14-case and 529-test results, static/tree verification, and the final + 23:42Z read-only host snapshot. The current external log ended after survey + target 26/47, while the guard still reported a resource wait and no final + pool, manifest, gates artifact, or runner verdict existed. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all pass. + Exact Git tree objects for both all `microcosm-build/src` and its + `us_runtime` subtree match all-build-suite checkpoint `a5be536f`. Since that + checkpoint only the already-rerun three-line binding regression and journals + changed. +- Reran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 files in one + process under the owner 12 GiB/20 ms guard. The run passed with 1.459 GiB + maximum observed per-process RSS. +- Re-traced bad commit `33bf52fe`, the current generic opt-in default, + owner-derived early/late selections, receipt emission, and terminal + validation. The old global binding preceded assignment; the current path + scopes all QRF regime behavior and evidence to the immutable nine-target + registry without weakening exact binding or the legacy count invariant. +- Independently audited the regression surface and found the exact synthetic + host target, real banked wide-family producer, unchanged mixed-family draw, + and strict rehashed assigned-family binding already covered. No duplicate + test edit is warranted. +- Ran the exact synthetic host-target case, the real banked wide-family case, + and all 12 QRF structure mutations under the owner 12 GiB/20 ms guard: 14 + passed with 0.541 GiB maximum observed per-process RSS. The only warning was + joblib's logical-core fallback. +- Reopened the committed journal for the 23:08Z continuation, preserved both + untracked owner artifacts, selected the GitNexus debugging workflow, and + began independent source, history, and regression audits of the supplied + binding failure. +- Refreshed `FINAL_REPORT.md` with this continuation's independent diagnosis, + current 14-case and 529-test results, static/tree verification, tooling + boundary, and exact non-terminal host status. +- Performed a final read-only check of the external host directory. The mutable + build log had been truncated to zero, the guard continued resource waits, and + no pool, manifest, gates, or runner-exit artifact existed; no after artifact + is accepted. +- Ran repository-wide Ruff lint, Ruff formatting checks on all 15 Python files + changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all pass. + The exact Git tree objects for both `microcosm-build/src` and its `us_runtime` + subtree match checkpoint `a5be536f`, where all 225 build-package test files + passed. Since that checkpoint only one three-line binding-test extension and + journals changed; the extended file is included in the current 529-test run. +- The prescribed `uv run --no-sync` lint attempt was blocked before execution + by the sandboxed user-wide uv cache. The same lock-synced worktree Ruff binary + was invoked directly from `.venv` and passed; no dependency or lockfile was + changed. +- Ran all five directly affected ordinary-transfer, multispine-serialization, + stacked-spine, pool-tool, and H5 files in one guarded process. Collection + confirmed 529 cases; pytest reached 100% with exit zero and no failures, and + peak observed per-process RSS was 1.617 GiB. +- Reconstructed the failure at `33bf52fe`: global QRF regime provenance was + attached to every imputed record and validated before the assigned-spec + branch, so the real bounded `puf_tax_itemization__batch_1` record conflicted + with the unassigned target's public-family expectation. +- Enumerated the immutable registry and canonical selectors at runtime: the + selected surface is exactly two early and seven late targets, and + `person/puf_tax_itemization/taxable_interest_income` is not assigned. +- Independently audited the existing regressions. They cover the exact forged + unassigned receipt, a real banked 15-target producer, unselected output + equivalence, and strict rehashed family binding including a plausible + `__batch_1` alias on an assigned target. +- Ran those 14 decisive cases under the owner-provided 12 GiB/20 ms guard; all + passed with 0.568 GiB maximum observed per-process RSS. The only warning was + joblib's logical-core fallback. +- Reopened the owner-requested ACS QRF binding audit and recorded the exact + starting revision, current uncertainty, and protected untracked inputs before + changing executable code. +- Refreshed `FINAL_REPORT.md` with the current independent diagnosis, complete + three-commit runtime correction, decisive regression coverage, guarded + 14-case and 529-test results, static/tree checks, GitNexus limitation, and + exact remaining host-only boundary. +- Performed the final external directory check read-only. The mutable retry had + rebuilt the exact bounded taxable-interest record, all 47 survey targets, + housing 1/1, and several late-transfer groups without the old traceback, but + remained active with no runner exit, final pool/manifest/gates artifacts, or + Microcosm revision binding. No after artifact is accepted. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all + passed. Exact Git tree objects for both `microcosm-build/src` and its + `us_runtime` subtree match checkpoint `a5be536f`, where all 225 build-package + test files passed. +- Ran all 529 tests in the five directly affected ordinary-transfer, + multispine-serialization, stacked-spine, pool-tool, and H5 I/O files in one + guarded process. Pytest reached 100% with exit zero and no failures; peak + observed per-process RSS was 1.523 GiB. Output contained the known pandas + fixture-fragmentation warnings and joblib logical-core fallback only. +- Ran the exact synthetic host-target regression, the real banked 15-target + producer regression, and all 12 fully rehashed QRF structure mutations under + the supplied guard. All 14 cases passed; peak observed per-process RSS was + 0.550 GiB. The only warning was joblib falling back from unavailable physical + core metadata to the logical-core count. +- Reconstructed the exact historical failure at `33bf52fe`: global regime + provenance reached every record; both early and late receipt builders + attached it for every imputed target; and the early validator called strict + QRF record binding before consulting calibration assignment. The traceback + line numbers exactly identify its unsplit-versus-`__batch_1` family mismatch. +- Enumerated the live immutable registry and every canonical selector. The + selected surface is exactly two early and seven late targets across the + assigned model-required, source-operator, and adult-care families; + `taxable_interest_income` is absent. Generic and multispine callers default + to no regime work, while canonical stacked producers pass only that exact + registry-derived selection. +- Audited producer provenance, receipt attachment, and terminal validation on + both early and late paths. Unassigned records have empty regimes, their + receipts omit QRF evidence, and validators reject forged evidence before + exact selected-family binding while still enforcing all four transfer + counts. +- Audited the decisive regression trio: the exact synthetic unassigned target, + the real banked 15-target family, and the fully rehashed plausible in-range + family forgery. Together they distinguish the bad global behavior from both + the final scoped fix and the intermediate permissive family matcher; no + material binding regression is missing. +- Followed the GitNexus debugging workflow. Analysis built a fresh 112 MiB + local graph, but the sandbox denied its registration through + `~/.gitnexus/registry.json`, so query/context could not resolve this repo. + The generated cache was removed; direct source, history, and regression + tracing independently established the same call path. +- Checked the active host retry read-only. It had passed survey 47/47 and + housing 1/1 and was partway through primary-QRF checkpoints without the old + traceback, but it remained mutable, had no runner exit marker or final pool, + manifest, or gates files, and carried no Microcosm revision binding. Host + success is not claimed. +- Reopened the committed root journal before any implementation or test work, + read `CLAUDE.md` and the GitNexus debugging instructions in full, inventoried + the existing correction/report history, and preserved both untracked owner + artifacts. +- Independently reproduced the historical call order at `33bf52fe`: every + record received QRF regimes and receipt evidence, and line 4512 validated the + unassigned target before spec lookup, causing line 4310 to reject its real + bounded family. A permissive batch alias would still have left an + eight-versus-fifteen regime-order mismatch. +- Audited the current producer, receipt, and validator paths. The only + production opt-ins derive from the immutable nine-target registry; generic + transfer defaults empty; `taxable_interest_income` receives no regimes or + QRF evidence; and both early and late validators reject forged unassigned + evidence before exact assigned-record binding while preserving four-count + accounting for every target. +- Confirmed that the complete audited runtime correction spans `176c60fc`, + `887df056`, and `94b7aecb`. Both `microcosm-build/src` and its `us_runtime` + subtree have the exact Git tree objects recorded at all-build-suite + checkpoint `a5be536f`; only journals and the later three-line strict-family + regression differ after that checkpoint. +- Audited existing regression coverage. The exact synthetic host receipt, a + real banked 15-target producer, and a fully rehashed plausible `__batch_1` + forgery on an assigned target jointly cover the failure and unsafe + workaround. Independent read-only runs passed the three scope-flow tests and + all 14 decisive binding cases; none is skipped or xfailed. +- Followed the GitNexus debugging workflow. Local analysis generated a 112 MiB + graph, but the sandbox denied registration through + `~/.gitnexus/registry.json`, so query/context could not target this repo. The + generated repo-local cache was removed; direct source and commit tracing + established the execution path without leaving residue. +- Ran the exact synthetic host-target regression, real banked 15-target + producer regression, and all 12 fully rehashed QRF structure mutations under + the owner-provided 12 GiB/20 ms guard. All 14 cases passed, with a 0.562 GiB + maximum observed per-process RSS. +- Ran all 529 tests in the five directly affected transfer, multispine + serialization, stacked-spine, pool-tool, and H5 I/O files in one guarded + process. Pytest reached 100% with exit zero and no failures; maximum observed + per-process RSS was 1.444 GiB. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all + pass. Exact Git tree objects for `us_runtime` and all `microcosm-build/src` + match `a5be536f`, and no runtime, tool, project, or lockfile differs from that + all-build-suite checkpoint. +- Rechecked the externally owned host directory read-only. Its current mutable + log reaches all 47 survey targets and the one housing target—including the + bounded taxable-interest record—without the supplied traceback, but stops + before a terminal verdict. The guard still reports a resource wait and no + runner exit marker, pool, manifest, or gates artifact exists, so no after + artifact or frozen-battery success is claimed. +- Refreshed `FINAL_REPORT.md` with the current independent diagnosis, complete + three-commit runtime correction, existing regression coverage, guarded + 14-case and 529-test results, static/tree checks, GitNexus limitation, and + exact remaining host-only boundary. +- Reopened the committed root journal before implementation or test changes, + reread `CLAUDE.md` and the GitNexus debugging workflow, inspected worktree + status and recent history, and preserved both untracked owner artifacts. +- Loaded `CLAUDE.md` and the GitNexus debugging instructions, inspected branch + status and recent history, preserved both untracked owner artifacts, and + recorded the newly supplied host traceback as the active continuation before + implementation or test changes. +- Reproduced the historical control flow directly at `33bf52fe`: the traceback + line numbers are exact, the 15-target family is bounded at width eight, and + unconditional evidence validation compares the real `__batch_1` record with + the unsplit family before looking up assignment. Accepting the alias alone + would next expose an eight-versus-fifteen regime-target-order mismatch. +- Audited the current producer and validator path. The two canonical stacked + callers select only the two early and seven late immutable calibration specs; + transfer regime work and provenance, receipt attachment, and terminal + validation independently use that selection. `taxable_interest_income` + retains its legacy QRF draw and bounded record but has empty regimes and no + QRF receipt evidence. +- Confirmed the correction spans `176c60fc`, `887df056`, and `94b7aecb`. + Current `us_runtime` and the entire `microcosm-build/src` tree have the same + Git tree objects as all-build-suite checkpoint `a5be536f`; the only later + nonjournal tracked change is the test-only in-range family forgery at + `21a48ba5`. +- Audited the regression surface independently. The exact synthetic unassigned + target and real banked wide-family tests fail under the old behavior, while + the fully rehashed assigned `__batch_1` alias fails under the intermediate + permissive matcher. All three pass on the current tree, and no material unit + gap remains for the reported boundary. +- Followed the GitNexus debugging workflow. A fresh 112 MiB local graph was + generated, but sandbox denial of the global registry write prevented normal + registration and the graph parser omitted the two large suspect runtime + modules. Query and context attempts therefore could not trace the raise-site + symbols; direct source/history tracing supplied the exact path. The generated + repo-local index was removed, leaving no tracked or untracked residue. +- Checked the cited host directory read-only. Its mutable `build.log` is now + zero bytes, `guard.log` still records resource waits, and no pool, manifest, + gates artifact, or terminal marker exists, so it cannot establish an after + verdict for the current revision. +- Ran the exact synthetic host-target regression, the real banked 15-target + producer regression, and all 12 fully rehashed QRF structure mutations: all + 14 cases passed under the owner 12 GiB/20 ms guard with a 0.568 GiB maximum + observed per-process RSS. The only warning was joblib's logical-core + fallback. +- Ran all five directly affected transfer, multispine serialization, stacked + spine, pool-tool, and H5 files in one guarded process. Pytest reached 100% + with exit zero and no failures; a separate collection bound that run to 529 + tests. Peak observed per-process RSS was 1.594 GiB, and warning summaries + were disabled to avoid the known fixture-fragmentation output. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, `git diff --check 33bf52fe^..HEAD`, and exact + Git tree-object comparisons for `us_runtime` and all `microcosm-build/src` + against `a5be536f`; all pass. The first format invocation supplied the + newline-separated paths as one zsh scalar and checked no file; the + null-delimited fail-fast rerun checked all 15 successfully. +- Refreshed `FINAL_REPORT.md` with the current independent diagnosis, complete + three-commit runtime correction, existing regression coverage, guarded + 14-case and 529-test results, static/tree checks, GitNexus limitation, and + exact remaining host-only boundary. +- Reopened and committed this root journal before changing implementation or + tests, reread `CLAUDE.md`, preserved the two untracked owner artifacts, and + recorded the exact supplied binding failure as the active continuation. +- Reproduced the historical failure chain at `33bf52fe`: universal regime + provenance reached every receipt; line 4512 validated the unassigned target + before the spec lookup; line 4310 rejected the bounded `__batch_1` record as + unequal to the unsplit canonical family. Accepting a batch alias would not + have fixed the over-broad behavior or target-order contract. +- Audited the current correction: generic transfer defaults regime selection + to empty; canonical early and late owners derive exact selections from the + two early and seven late immutable specs; producers emit QRF evidence only + for that selection; validators reject evidence before continuing past every + unassigned target and preserve strict four-count accounting. +- Confirmed `git diff --quiet a5be536f..HEAD` over build source/tests, tools, + specs, `pyproject.toml`, and `uv.lock`: no committed executable drift exists + after the prior complete build-suite checkpoint. +- Followed the GitNexus debugging workflow. The repository had no index; local + analysis generated its graph but could not register or query it because the + sandbox forbids writing `~/.gitnexus/registry.json`. The generated 112 MiB + repo-local cache was removed with `gitnexus clean`; direct source and history + tracing supplied the call chain instead. +- Reconciled independent source and regression audits. The source audit found + no canonical producer, serializer, or validator path that gives the host + target regime work or evidence. The regression audit confirmed both the + exact synthetic target and real banked wide-family cases fail on the old + behavior, then identified one strict selected-family binding gap. +- Added an assigned-target, fully rehashed `__batch_1` mutation to the QRF + structure parameterization. It protects exact record-family equality from a + previously considered permissive bounded-family matcher without changing + runtime code. +- Ran the eight focused scope/serialization tests plus all 12 QRF structure + mutations: 20 passed under the owner-provided 12 GiB/20 ms guard with a + 0.586 GiB maximum observed per-process RSS. The only warning was joblib's + logical-core fallback. The initial guard invocation did not start because + the untracked script lacks an executable bit; invoking it through the exact + environment's Python succeeded without modifying the owner artifact. +- Ran all 529 tests collected from the five directly affected files in one + process under the 12 GiB/20 ms guard. Every test passed; maximum observed + per-process RSS was 1.531 GiB. Warning summaries were disabled for the run; + the same files previously emitted known fixture-construction fragmentation + warnings. +- Ran repository-wide `ruff check .`, `ruff format --check` on all 15 Python + files changed since `33bf52fe^`, and `git diff --check 33bf52fe^..HEAD`; all + pass. Confirmed no build runtime, tool, spec, project, or lockfile differs + from all-suite checkpoint `a5be536f`; only the added regression changes that + checkpoint's test tree. +- Performed a final read-only host-status check without controlling the external + runner. The mutable retry log that had previously crossed all early survey + and housing targets was truncated to zero bytes at 20:42Z; `guard.log` still + reported a resource wait, and the directory contained only those two logs. + No terminal host verdict or revision-bound artifact is available. +- Refreshed `FINAL_REPORT.md` with the current root cause, assigned-only + correction, new in-range family regression, guarded 529-test verification, + lint/diff results, continuation commits, and exact remaining host boundary. +- Reopened the committed progress journal before changing implementation or + tests, reread `CLAUDE.md`, and recorded the supplied failure as the active + continuation target. +- Re-traced the old raise site and confirmed the failure sequence: the + 15-target family was split at width eight, the target record identified its + bounded `__batch_1` family, and the old global validator compared that record + with the unsplit canonical family before checking whether the target was + assigned. Merely accepting the batch alias would have exposed the subsequent + eight-versus-fifteen target-order mismatch. +- Audited the current source and repository-wide selector call sites. Generic + ACS transfer regime evidence defaults to empty; the two canonical stacked + callers are the only production selectors and derive exact `(entity, + target)` pairs from the nine post-transfer calibration specs on their live + surfaces. Receipt producers and validators independently reconstruct the + same selected set. `taxable_interest_income` is absent from it. +- Reconciled two independent read-only audits of the current call path and the + failing-to-fixed history. Both found no remaining canonical scope leak and + no executable drift after `a5be536f`; only the two root journals differ from + that all-build-suite checkpoint. +- Followed the GitNexus debugging skill as far as this session permits. Graph + query/context tools and a local index are unavailable, and the installed CLI + status probe produced no output and was stopped. Direct source, history, and + repository-wide call-site tracing established the same execution path. +- Ran eight focused regressions covering the exact synthetic host boundary, + the real banked 15-target family, default wide-family behavior, selected-only + provenance, complete selected siblings, mixed-family output identity, and + both generic serializers. All passed under the owner-provided 12 GiB/20 ms + guard; peak observed per-process RSS was 0.559 GiB. The only warning was + joblib falling back from an unavailable physical-core count to the logical + core count. +- Ran all 528 tests in the five directly affected ordinary transfer, + multispine serialization, stacked spine, pool-tool, and H5 files in one + process under the 12 GiB/20 ms guard. All passed with a 1.494 GiB maximum + observed per-process RSS. Output contained the same joblib logical-core + fallback and 2,313 known pandas fragmentation warnings from stacked-spine + fixture construction. +- Ran repository-wide `ruff check .`, `ruff format --check` on all nine Python + files changed by the scoped correction, and `git diff --check + 33bf52fe^..HEAD`; all passed. `git diff --name-status a5be536f..HEAD` lists + only `FINAL_REPORT.md` and `PROGRESS.md`, proving no executable drift from the + prior complete 225-file `microcosm-build` pass. +- Reconciled the third independent regression audit. It confirmed that the + exact synthetic canonical-validator case and the real banked producer case + jointly cover the reported boundary; they remain separate because a + test-authority receipt must not acquire canonical production authority. +- Rechecked the externally owned host directory without controlling it. The + reused log reaches all 47 early survey targets, housing 1/1, and the late DAG + without the reported traceback, while the directory still has no exit + marker, pool, manifest, or gates artifact and the log does not bind a + revision SHA. The external guard continues to report other live work, so no + host certification is claimed. +- Refreshed `FINAL_REPORT.md` with the current diagnosis, assigned-only scope, + regression structure, guarded verification, continuation commits, and exact + remaining host boundary. +- Reopened the committed progress journal before implementation work for the + current owner continuation and recorded the supplied failure as the active + verification target. +- Confirmed the supplied traceback's exact pre-fix call order at `33bf52fe`: + `validate_stacked_gap_fill_receipt` invoked + `_validate_acs_imputed_pattern_evidence` on every early target before checking + whether the target had a declared calibration spec. The real bounded record + family `puf_tax_itemization__batch_1` could not bind to the canonical unsplit + family, producing the reported error at line 4310. +- Confirmed the current tree scopes regime detection, fitted/chain checking, + per-record provenance, receipt attachment, and terminal validation through + the exact immutable post-transfer calibration selection. The host target is + absent from that selection and cannot carry QRF evidence. +- Ran seven focused scope tests. The exact synthetic host receipt, real banked + wide-family integration, default wide-family behavior, selected-only + provenance, mixed-family output equivalence, and both generic JSON + serializers all passed. +- Ran all 528 tests in the five directly affected files in one process under + the owner-provided 12 GiB/20 ms guard. All passed; peak observed per-process + RSS was 1.531 GiB. The only output was 2,313 known pandas fragmentation + warnings from stacked-spine test fixture construction. +- Re-ran repository-wide `ruff check .`, formatting checks on all nine Python + files touched by the scoped correction, and `git diff --check + 33bf52fe^..HEAD`; all passed. Only `FINAL_REPORT.md` and `PROGRESS.md` differ + from `a5be536f`, whose complete 225-file `microcosm-build` run passed, so the + currently verified executable tree is identical to that full-suite tree. +- Reconciled three independent read-only audits. The source audit found no + canonical path that can opt the host target into regime work or receipt + evidence; the regression audit confirmed the exact synthetic and real banked + tests materially cover the reported leak; and the host audit found no newer + terminal verdict. +- Audited the later host retry without controlling it. Its reused build log + reaches survey 47/47 and housing 1/1, including the bounded taxable-interest + record, with no traceback. It has no terminal exit marker, pool H5, manifest, + or gates artifact, and the log does not bind a revision SHA; it is progress + evidence only. +- Updated and committed `FINAL_REPORT.md` with the current diagnosis, + correction, regression coverage, guarded verification, and precise remaining + host boundary. +- Attempted the GitNexus debugging workflow. No graph-query tools or local index + are available in this session; the installed CLI status command produced no + output and was stopped. Direct source and history tracing established the + failing and corrected call paths without changing the tree. +- Reopened the committed journal at the start of the owner continuation, + preserved both untracked host artifacts, and began an independent audit of + the existing scoped-binding correction. +- Re-traced the failing 15-target `puf_tax_itemization` family from its + certified eight-target split through serialized QRF evidence and terminal + validation. The host record used `puf_tax_itemization__batch_1`, while the + old global validator expected the unsplit family; merely accepting the batch + alias would next have failed its eight-versus-fifteen target-order check. +- Confirmed the current correction keeps exact record binding for the nine + declared calibration targets while defaulting regime detection, + verification, provenance, and receipt evidence off elsewhere. The exact + `taxable_interest_income` synthetic binding regression, a real banked + 15-target transfer, selected-only provenance, default-wide-family behavior, + generic serialization, and mixed-family output equivalence all pass (six + focused tests). +- Followed the GitNexus debugging workflow: a local graph was generated and + reported current at commit `40b76f6b`, but its final registry write and + query/context registration were sandbox-blocked at + `~/.gitnexus/registry.json`. Direct graph status, source, history, and test + tracing independently converged on the same call chain; the generated local + cache was removed afterward. +- Re-ran all five directly affected files—ordinary transfer, multispine + serialization, stacked spine, pool tool, and H5 I/O—in one fresh process: + 528 passed, zero skipped, zero failed/errors. The only output was 2,313 + pandas fragmentation warnings from stacked-spine fixture construction. +- Re-ran repository-wide `ruff check .`, format checking on all nine + continuation-touched Python files, and `git diff --check 33bf52fe^..HEAD`; + all pass. Confirmed no executable file differs from the prior completion + commit `a5be536f`, whose all-225-file `microcosm-build` run was green. +- Observed, without controlling it, an externally owned exact host retry. It + rebuilt all 47 survey targets plus housing 1/1, including + `person/puf_tax_itemization__batch_1/taxable_interest_income`, with no + traceback. The build log then stopped changing; the external guard still + reported two live matching build processes and the runner had written no + terminal exit marker. This is progress through transfer, not proof that the + stacked receipt invariant or frozen battery passed. +- Read `CLAUDE.md`, the adjudication and all assigned machine-readable rows, + citation registry entries C04-C08, and every cited source range. +- Built and measured the canonical off-chain 1% before artifact with sample + seed 578, clone fraction 1, clone seed 578, no predecessor digest, and no + pending-chain file mutation. +- Added a declared nine-target positive-leg policy. It preserves unemployment + and disability carriers, matches the other assigned positive carrier + margins, maps mutable positive amounts to exact reference support at the + frozen five quantiles, and byte-preserves negative, negative-zero, + zero-weight, and nonmutable values. +- Bound production application to exact ASEC clone-0 reference rows, ACS + clone-0 recipients, and transferred-null mutable cells. Adult-care additions + are limited to qualifying people and one candidate per empty tax unit; + positive weeks additions require positive unemployment compensation. +- Reused the adult-care qualifying predicate for both calibration and the + final reconciliation, which must be a verified no-op after calibration. +- Recorded and validated exact donor-support QRF regimes only for the + explicitly owner-selected ordinary and banked ACS targets, without changing + fit seeds, folds, estimator counts, or draw behavior. Ordinary callers + default to no regime work or provenance. +- Added schema-v2 calibration receipts with explicit terminal-versus- + generation verification boundaries. Terminal validation independently + replays live masks, row identities, output bytes, weights, carrier metrics, + conditional quantiles, and coupled constraints; adversarial fully rehashed + scope and diagnostic forgeries are rejected. +- Changed the two pinned SIPP chunk readers to streaming type inference. This + leaves downstream explicit numeric coercion and locked output facts intact + while reducing the 3.73 GB donor-test peaks from above the safety ceiling to + 0.49 GiB and 0.53 GiB. +- Verified all 225 `microcosm-build` test files green before the host + continuation and again on the final scoped-fix tree, split into fresh pytest + processes where needed. Also verified the unchanged sibling packages before + the continuation: `microcosm-fit` (93 passed), `microcosm-calibrate` (201 + passed), `microcosm-frame` (294 passed, 36 skipped), and `microcosm-data` + (275 passed, one skipped). +- Ran repository-wide `ruff check .`, touched-file `ruff format --check`, and + `git diff --check` successfully. Repository-wide `ruff format --check .` + still identifies 49 pre-existing, mostly unrelated files; none was + reformatted as part of this lane. +- Kept final verification processes below 10 GiB with a 10 GiB/20 ms guard. + During diagnosis, the earlier 13.5 GiB/250 ms guard observed one rapid SIPP + parser spike at 15.424 GiB before terminating it; the reader fix and all + successful reruns remained far below the cap. This exception is retained in + the journal rather than concealed. +- Recorded the owner-provided host failure at the exact + `_validate_acs_imputed_pattern_evidence` raise site. The failing target is + outside this lane's assigned rows, so shared gap-fill behavior must be + restored before another artifact build. +- Traced the complete failure path. `transfer_acs_inputs` intentionally splits + families wider than eight targets and records the bounded family name, while + the new terminal validator binds every record to the canonical unsplit + family and canonical target list. Merely accepting the batch suffix would + expose a second target-order mismatch and would retain the out-of-scope + runtime checks. +- Confirmed that value calibration itself already filters the exact declared + stage/entity/family/target triples and mutates only ACS clone-0 cells that + were null before transfer and nonnull afterward. The leak is limited to the + globally enabled QRF regime detection/verification and receipt surface. +- Built a fresh local GitNexus graph as directed by the debugging workflow. + The sandbox prevents registering it in the user-wide GitNexus registry, so + query/context calls could not consume it; source, history, and test tracing + independently established the caller and data flow. +- Added a failing-first regression for the exact host target + `person/puf_tax_itemization/taxable_interest_income`. It proves undeclared + batched QRF evidence is rejected while the evidence-free legacy count + receipt validates unchanged. +- Scoped QRF regime detection, fitted-result checking, banked-chain checking, + record provenance, stacked receipt attachment, and terminal validation to + the calibration specs on the current transfer surface. Unassigned target + records carry empty regime tuples, and both generic ACS receipt serializers + omit that empty opt-in field to preserve their legacy JSON schema. +- Focused post-fix verification is green across ordinary/banked transfer, + selected and unassigned stacked binding, real late execution, both generic + serializers, forged batch aliases, and the original wide-family case. +- Added failing-first early and late regressions after independent review found + that unassigned receipts skipped the four legacy transfer counts. Count + presence, nonnegative integer types, activation accounting, and residual + accounting now validate before either the assigned or unassigned branch. +- Added a banked integration regression that runs the full 15-target + `puf_tax_itemization` family beside selected unemployment compensation. Its + real `taxable_interest_income` record is `__batch_1`, has no regimes or QRF + receipt, while the selected unemployment record retains both. A complete + selected sibling is kept in a partially active selected fit so receipt + target order stays exact without mutating its complete recipient cells. +- Closed the final review findings: canonical validators require the complete + four-count block rather than accepting wholesale stripping; canonical gap + fill rejects non-default fit widths just as the late DAG already does; and + complete selected siblings are restored only when every family target is + selected. A mixed selected/unselected regression proves the opt-in does not + change the unassigned draw. +- The complete build-package rerun reached the canonical stacked H5 loader and + correctly rejected its synthetic late-transfer targets because the fixture + supplied only `residual_null_rows`. Updated that fixture to the internally + consistent four-zero target-count block; production code was not relaxed. +- Completed the post-fix rerun after that fixture correction. Every directly + affected transfer/stacked/pool test file and every remaining build-package + test file is green. A repository-wide scan found no other canonical target + receipt fixture with a partial four-count block. + +## Next + +1. Host certification still requires a revision-bound off-chain 1% retry with + a terminal verdict and final pool/manifest/gates artifacts. Do not publish or + mutate the pending logbook chain as a side effect. +## Historical journal imported from `origin/main` — 2026-08-22 + +The following `one-target-surface` state/done/next record is preserved from +the merged `origin/main` history. Its branch-state claims and remaining-next +instructions describe that completed source lane at the time they were written; +they are not current instructions for `battery-pkg3-two-part`. + # Progress: one US target surface ## State diff --git a/_LANE-NOTES.md b/_LANE-NOTES.md index ab89acae..eb26cf90 100644 --- a/_LANE-NOTES.md +++ b/_LANE-NOTES.md @@ -1,3 +1,563 @@ +# Battery package 3 lane notes + +## Main/F0 merge continuation — 2026-08-22 02:09Z + +- Starting tracked revision: `c22e5d37`; local `origin/main`: `b4dfa0e7`. +- This continuation owns only the main merge, the F0 home for the existing + post-draw calibration policy declaration, its spec/code identity test, and + the resulting closed-world anti-rot updates. It does not run a host build, + change a comparator/gate/threshold, certify an artifact, publish a release, + or merge PR #742. +- Main's generated F0 bundle supersedes `specs/us_imputation_lineage.yaml`, so + the old file will remain deleted. The policy itself must survive by becoming + a closed, typed exact variant at the authored imputation-model location. +- Earlier notes below remain source-cited history. Their branch-state and + handoff language is not current for this continuation. +- Salvage audit: `ea21353d` has parent `bb94f789` and matches the current + tracked merge worktree byte-for-byte. It preserved literal conflict markers + and accidentally tracked `.codex-memory-guard.py` plus + `_BUILD-FAILURE-1PCT.txt`; it contains no F0 schema/spec/anti-rot edit. The + live automatic merge of `tools/build_us_multispine_pool.py` is retained + because it carries main's F0 adapter path and this branch's omission of empty + `target_regimes`. No salvage path was checked out or cherry-picked; the + marker snapshots, old YAML, and accidental diagnostic additions were + discarded while the owner files remain untracked and untouched. +- Salvage audit, second and third snapshots (2026-08-22 continuation): + `e2402fd4` and `abcf7dc1` also have parent `bb94f789`. The worktree at + recovery time already equaled `abcf7dc1` for every tracked path (verified + with `git diff abcf7dc1 --stat`: the only differences were the two + salvage-only debris files), so the third snapshot's merge resolution, F0 + policy port, and anti-rot edits were recovered in place — nothing was + checked out or cherry-picked. Taken: the staged `b4dfa0e7` merge + resolution; the closed `regime_gated_qrf_model` exact variant with the + `post_transfer_calibration_policy_v1` payload + (`packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json`); + the authored declaration and its generator + (`packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml`, + `tools/us_bundle_generation/imputation.py`); the ninth + `post_transfer_calibration` authority component and version-11 binding + (`packages/microcosm-build/src/microcosm/build/spec_engine/stacked_authority_semantics.py`, + `.../us/spec/battery.yaml`); and the walked anti-rot literals + (field_usage/inventory/coverage/test pins). Discarded: + `.codex-memory-guard.py` (carpool tooling debris) and + `_BUILD-FAILURE-1PCT.txt` (stale r3 continuation note; its 1% failure was + already closed on-branch by `a932974f`..`c22e5d37`). +- Chain site the salvage had not reached: the loader golden vector and every + bundle `spec_sha256` pin move on this merge, and the generating mechanism was + verified by single-file bisection, not assumed. `spec_sha256` hashes only the + spec envelope — country, manifest schema version, normative projections, and + resolved bindings (`spec_engine/loader.py:404-418`, `canonical.py:299-319`); + the schema-set receipt lives in the grammar receipt outside that envelope + (`loader.py:227-229,401`), which is why main's `sources.schema.json` property + additions moved no pin on main. The resolved bindings embed the selected seed + protocol's wire form, whose kernel attestations digest the bytes of the + attested kernel modules (`spec_engine/seeds.py:40-71,92-97,189-199,309-370`), + and the attested list includes `microcosm.build.us_runtime.acs_transfer`, + `weeks_unemployed`, `puf_qrf_chain`, and `microcosm.fit.qrf` — modules this + branch's calibration repair changed. Reverting each of this continuation's + schema/semantics/tool edits one at a time left the synthetic golden value + unchanged at `b1ab6ab0…`, confirming the branch's attested-kernel edits (not + the shared-schema edit) are the mover, exactly the legitimate identity + movement `CLAUDE.md` documents. Pin re-cut in + `packages/microcosm-build/tests/test_spec_engine_loader.py::test_semantic_hash_has_golden_vector_and_surface_separation`. +- Verified unaffected: the UK gate-battery mirrors recompute from the live UK + gates manifest, not the schema registry — all ten + `packages/microcosm-build/tests/test_gate_battery_contract_pins.py` tests + pass on this tree with the imputation-schema variant present, so the + `microcosm-data` contract pins from `b4dfa0e7` stand here without a re-cut. + +## Second merge: `origin/main` at `2aa96795` — 2026-08-22 headless run + +- The first merge (with the recovered port, the re-cut loader golden vector, + and the changelog fragment) is commit `e66074ad`; the second merge is + commit `e0947020`. `origin/main` had advanced past the staged + `b4dfa0e7` MERGE_HEAD by `1d066e5d` (#733 review fixes), `d70ea39c` + (post-#735 UK spec / gate-battery digest re-cut), and merge `2aa96795` + (#733 UK FRS 2024-25 retarget). +- Sole textual conflict: the UK entry of `EXPECTED_RESOURCES` in + `packages/microcosm-build/tests/test_spec_engine_country_bundles.py`. Both + parents' pins were computed on trees missing the other side's envelope + movers — this branch's `aa32c4c9…` includes the stacked-authority + version-11 binding but not #733's UK sources/runtime retarget; main's + `e12a2cb8…` the reverse. `spec_sha256` hashes the spec envelope — country, + manifest schema version, normative projections, resolved bindings + (`spec_engine/loader.py:404-418`) — so the union pin was recomputed fresh + via `load_bundle("uk")`: `bb711069…`, a third value, as required. +- BE (`bf022118…`) and US (`d3de6760…`) recomputed identically on the union: + the `b4dfa0e7..2aa96795` range touches no BE- or US-side envelope input. + Its only shared-schema edit, `sources.schema.json`, sits in the grammar + receipt outside the hashed envelope (`spec_engine/loader.py:227-229,401`), + the mechanism already proven when main's property additions moved no pin. +- Main's re-cut UK gate-battery digests (`uk/gates.json`, the + `microcosm-data` contract pins in `contract.py`/`test_contract.py`) merged + clean — this branch touches neither side of them — and the ten + gate-battery pin tests plus `test_contract.py` pass unchanged on the + union, inside the 339-test affected run. +- Union-tree receipts before the full suite: `tools/spec_engine_coverage.py + --check` reports 41,471/41,471 configuration fields (main's 41,379 plus + this branch's 92 policy fields) and 40/40 inventory checks with no drift; + the loader golden vector holds (main's range touches neither + `loader.py` nor `canonical.py`); repository-wide Ruff and + `git diff --check` pass. +- Wheel gate (the merge moves `microcosm-build/pyproject.toml` and packaged + spec data): all five shard wheels build; a clean venv installed from those + wheels under the exported lock constraints imports every shard from the + venv prefix with `policyengine_us` absent; + `tools/spec_envelope_digests.py be uk` runs from the installed wheels; and + wheel-venv `load_bundle` identities reproduce BE `bf022118…` / UK + `bb711069…` byte-for-byte, proving the authored spec and schema files ship + in the wheel (`packages/microcosm-build/pyproject.toml` data inclusion). + +## Build-shard merge-union guards — 2026-08-23 headless continuation + +- The first complete build-shard run reached 100% and exposed three + deterministic union misses. The branch-side serializer omitted an empty + opt-in `target_regimes` field by routing dataclasses through mappings; main + replaced `dataclasses.asdict()` with direct field walking so immutable + `MappingProxyType` authority records serialize. The merged mapping path had + the omission filter, but the direct dataclass path did not. The union now + applies the same filter during direct walking + (`tools/build_us_multispine_pool.py:3826-3847`), preserving both intents. +- Main's authored-imputation SHA audit treated every `*sha256` field as an + external-asset pin. The F0 port legitimately authors one derived policy + identity digest. The audit now remains closed: its only non-asset allowance + is the exact + `models/regime_gated_qrf/post_draw_calibration/sha256` path, while the two + existing asset pins remain exact + (`packages/microcosm-build/tests/test_us_spec_bundle.py:707-770`). +- Direct import-graph enumeration reaches 66 classified US runtime modules; + the delta from main's 65 is exactly the branch-added + `post_transfer_calibration.py`. The cardinality pin now says 66, while the + existing required-module, retired-module, unclassified-module, and + source-spine-blind checks remain unchanged and pass + (`packages/microcosm-build/tests/test_us_spine_blindness.py:3270-3315`). +- All three exact deterministic tests pass, as do touched-file Ruff, + formatting, and whitespace checks. GitNexus query/context tools were not + exposed in this session, so parent-source comparison, direct import-graph + enumeration, and exact source reads supplied the debugging evidence. +- Two unrelated `test_us_trade_imdb_bulk.py` crash-publication cases timed out + while their child process imported `build_us_import_entry_margins.py`, before + reaching publication logic. They failed both in the complete shard and in + unchanged isolation under shared-host load. A one-thread, import-only probe + completed in 73.82 seconds while load was about 79, already beyond the + tests' unchanged 60-second bound. This lane will retry after contention + falls; it does not tune the timeout or any product threshold. + +## Third merge: `origin/main` at `055dcfaf` — 2026-08-23 + +- The shared remote-tracking ref advanced six commits while the complete build + shard ran: UK E8 CGT structure, salary sacrifice, and student-loan stages; + their review fixes; the US fiscal-refresh memory-canary isolation; and merge + `055dcfaf` (#740). This range does not touch the F0 imputation policy, its + anti-rot ledger, or the US spec envelope. +- Sole textual conflict: the UK entry of `EXPECTED_RESOURCES` in + `packages/microcosm-build/tests/test_spec_engine_country_bundles.py`, for the + same two-parent reason as the prior merge. This branch's `bb711069…` omits + E8; main's `1f163cbf…` omits the version-11 stacked-authority binding. Fresh + union-tree `load_bundle` results are BE `bf022118…`, UK `8bf62b6e…`, and US + `d3de6760…`; the conflict is resolved with the UK union value. +- Main's updated UK gate-battery and `microcosm-data` contract pins merged + cleanly. The new shared `sources.schema.json` variants remain in the grammar + receipt outside the hashed envelope (`spec_engine/loader.py:227-229,401`), + while E8's authored UK sources legitimately move only the UK envelope. +- Newest-union verification before committing the merge: + `tools/spec_engine_coverage.py --check` reports 41,471/41,471 fields and + 40/40 inventory checks; the focused policy identity, country bundle, field + usage, coverage, inventory, gate/data contract, serializer, authored-SHA, + and 66-module spine-blind graph tests all pass in one process. + +## Final newest-union verification — 2026-08-23 + +- Re-ran the two unchanged `test_us_trade_imdb_bulk.py` crash-publication + cases after shared-host load fell. Both pass with the existing 60-second + child-process bound and one-thread numerical-library environment. The + earlier failures were confined to child CLI import under contention; no + timeout, comparator, gate, or product threshold changed. +- Repository-wide `ruff check .` and `git diff --check` pass on the final + `055dcfaf` union. +- Four shard receipts, each in one pytest process with peak child RSS from + `resource.getrusage`: fit 93 passed, exit 0, 925,466,624 bytes (0.862 GiB); + calibrate 203 passed, exit 0, 481,968,128 bytes (0.449 GiB); frame 294 + passed / 36 skipped, exit 0, 6,971,260,928 bytes (6.492 GiB); data 275 + passed / one skipped, exit 0, 11,867,324,416 bytes (11.052 GiB). +- The complete build root first passed behaviorally in one process (6,248 + passed / 39 skipped) but reached 17,167,810,560 bytes (15.989 GiB). That + receipt violates this lane's `<15 GiB` constraint and is explicitly + rejected; it is not the resource evidence for completion. +- The authoritative build rerun enumerated all 262 `test_*.py` files and ran + them in 17 fresh pytest processes under a 12 GiB/20 ms hard guard. Every + batch passed without a split or guard intervention. The exact aggregate is + 6,248 passed / 39 skipped, exit 0, elapsed 6,966.45 seconds; maximum observed + RSS is 11,042,193,408 bytes (10.284 GiB). This reproduces the complete + single-process inventory while keeping the accepted receipt below 15 GiB. +- Refreshed the packaging boundary on the newest union. All five wheels built + offline using the writable cache. A brand-new offline venv was created but + could not resolve uncached third-party packages (`pytest`, then + `huggingface-hub`), so it was not used as evidence. The five new local + wheels were instead reinstalled without dependency resolution into the + existing clean, lock-constrained wheel venv from the earlier gate. With + `PYTHONPATH` removed and isolated mode enabled, all five `microcosm.*` + shards import from that venv, `policyengine_us` is absent, + `tools/spec_envelope_digests.py be uk` passes, and installed-wheel + identities are BE `bf0221184046428782e7628dfad9b1a420bcc90c76ee88f4df373abecabff9d9`, + UK `8bf62b6e47583da1bdad1b71be1e705f424e6e245880e90f4411aba57fa5eb93`, + and US `d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0`. +- A fresh `git fetch origin main battery-pkg3-two-part` remains blocked before + transport by sandbox DNS (`Could not resolve host: github.com`). The shared + local `origin/main` ref is `055dcfaf` and is an ancestor of HEAD. No push has + been attempted; the sole permitted push remains the final transport step. + +## Scope and frozen boundary + +This lane owns 16 adjudicated FIX-CANDIDATE checks: 13 in +`source_operator_two_part_calibration`, two in +`adult_care_post_reconciliation`, and one in +`model_required_targeted_calibration`. The assigned baselines and named +remedies are the rows in the read-only reference +`../microcosm-arm-split/experiments/battery_burndown/adjudication.json`. + +No comparator, band, threshold, seed, fold, or sample contract may change. All +builds in this lane are off-chain at `--sample-fraction 0.01` and +`--sample-seed 578`; they omit `--logbook-prev-row-digest` and do not touch +`logbook-pending-chain.txt`. + +## Post-transfer receipt failure #2: ordered capacity evidence + +### No-build reproduction + +- The SHA-pinned replay harness reads the assembled Frame checkpoint and the + unemployment-compensation and weeks-unemployed target-bank H5 files, + validates their file, identity, and raw-draw digests, reconstructs only the + native clone-0 vectors, and calls the calibration kernel and its strict + receipt validator. It performs no target fit, DAG execution, Frame write, or + build (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:1-6,74-112,115-250`). +- The stacked owner selects ASEC clone-0 reference rows, ACS clone-0 recipient + rows, transferred-null mutable rows, and—for weeks—only mutable rows with + positive unemployment compensation as allowed/addition candidates + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8960-8971,8995-9032`). +- The preserved replay contains 4,311 reference rows, 34,293 recipient/mutable + rows, 134 positive reference rows, 24 initial recipient positives (all + disallowed), and 32 positive-UC addition candidates. Its reference total is + `80,851,529.27715749`, recipient total is `79,926,522.10879111`, reference + positive mass is `2,762,659.3294707513`, and target positive mass is + `2,731,052.2627107087` (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:144-206,230-250`). +- At reproduction commit `4cc41652`, capacity generation reduces the 32 + candidate weights with masked `ndarray.sum`, producing + `85,676.23791782455`, while selection independently reduces the same + ID-ordered weights with `np.cumsum`, producing `85,676.23791782456` + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:463-493,817-880` at that commit). + The exact validator relationship `0 <= lower <= upper <= + expected_candidate_mass` therefore rejects the receipt by one float64 ULP, + `1.4551915228366852e-11`; every other relationship passes + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:1389-1490` at `4cc41652`). + +### Semantic adjudication + +- The invariant is correct and target-type agnostic: a selected prefix cannot + exceed its declared candidate capacity. No tolerance or target exception is + authorized. The generating defect is that capacity and prefix evidence are + derived by two reduction schedules for the same declared ordered carrier + set (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:321-327,463-493,817-891` at `4cc41652`). +- Weeks is a valid positive-carrier target. The ASEC source accepts only + integer `-1` or `0..52`, maps `-1` to zero, and defines the positive event as + an in-range integer above zero + (`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:791-800,1218-1222`). + Its dedicated QRF path rounds, clips, and revalidates predictions in that + domain; post-transfer amount mapping then selects only positive + reference-donor values and rejects support escape + (`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:911-983`; + `packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:588-626,702-705`). +- The semantics-preserving repair is one immutable ordered-prefix schedule per + candidate set, consumed by both capacity and selection. Rewriting only the + terminal cumulative value is invalid because `_nearest_prefix` requires a + nondecreasing vector for its lower-mass tie break and `searchsorted` + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:487-515`). + +### Two generating defects and the complete repair + +- `_PrefixSchedule` binds one immutable ordered-position vector to one + float64 cumulative-mass vector. `_nearest_prefix` consumes that schedule and + reports its terminal element as candidate mass + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:282-287,471-515`). + Commit `d7b12bab` constructs the amount-descending/ID removal schedule and + ID-ordered addition schedule once, so capacity and selection use the same + candidate endpoint (`post_transfer_calibration.py:844-872,891-928`). +- The SHA-pinned weeks replay therefore changes its candidate mass from + `85,676.23791782455` to the selection schedule's exact terminal + `85,676.23791782456`; upper minus candidate becomes zero and strict + validation succeeds. The harness now pins both the failing and repaired + relationship values rather than accepting an arbitrary validator failure + (`tools/reproduce_us_post_transfer_weeks_checkpoint.py:210-286`). +- The required cross-target replay found that `d7b12bab` still composed its + whole maximum from independently rounded partition endpoints. Both actual + child-support targets produced maximum `79,926,522.10879174` against + recipient total `79,926,522.10879111`, so only the exact + `maximum_attainable_mass <= recipient_total` relationship failed, by + `6.258487701416016e-07`. Expense partitions were + `71,696.09739141785 + 79,854,826.01140033`; received partitions were + `180,209.75664861224 + 79,746,312.35214312`. +- The complete generating repair declares the attainable carrier set once as + `fixed_positive | allowed_positive | zero_candidates`. It zero-masks that + set onto the already ordered recipient-weight vector, retaining the same + vector length and reduction topology used by `recipient_total`. For + nonnegative weights, the exact subset bound is therefore structural; the + maximum is neither a sum of rounded partition scalars nor a clamp + (`post_transfer_calibration.py:823-885`). +- Both SHA-pinned child receipts now have maximum exactly equal to recipient + total `79,926,522.10879111` while the historical partition sum remains + `79,926,522.10879174`; every strict relationship passes. The child harness + pins both targets' file/identity/raw hashes and requires the exact per-target + state, error, relationship, and floats on the red and green sides + (`tools/audit_us_post_transfer_child_support_checkpoints.py:1-80,91-207,210-302`). +- A proper-subset regression supplies weights for which compressed regrouping + yields `0x1.433526fbe1946p+48`, `0.0625` above recipient total + `0x1.433526fbe1945p+48`. The same-topology union yields the recipient value + exactly and validates for every late `match_reference` declaration. Separate + regressions cover the production weeks candidate bytes, independently + rounded whole partitions, and the symmetric removal path + (`packages/microcosm-build/tests/test_us_post_transfer_calibration.py:544-753`). +- The validator is unchanged: it still requires exact + `maximum_attainable_mass <= recipient_total` and exact + `0 <= lower <= upper <= candidate_mass`; its pre-existing approximate + partition-additivity and boundary checks were not adjusted + (`post_transfer_calibration.py:1457-1529`). No tolerance, threshold, band, + gate, or target exception changed. + +### Complete late-transfer target audit + +The registry contains exactly seven late targets; six share the repaired +`match_reference` branch and one bypasses carrier selection by preserving +recipient carriers +(`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:208-258,786-812,840-932`). + +| Late target | Source semantics / observed 1% shape | Carrier verdict | +| --- | --- | --- | +| `pre_subsidy_care_expenses` | Nonnegative monetary care expense. ACS reconciliation restricts it to qualifying people and at most one carrier per tax unit; the late owner admits existing qualifying positives and one stable zero candidate per empty unit (`acs_transfer.py:660-739,1277-1299`; `stacked_spine.py:8728-8746,8977-8986`). | Covered: constrained `match_reference`; not a count. Its proper attainable subset uses the same whole-union mechanism and proper-subset regression. Host had not reached it. | +| `child_support_expense` | Exact nonnegative annual `CHSP_VAL` carry (`child_support.py:166-201`); QRF reported six distinct donor values. | Covered: monetary `match_reference`. Its pinned checkpoint fails at `d7b12bab` and passes the complete union repair. | +| `child_support_received` | Exact nonnegative annual `CSP_VAL` carry (`child_support.py:166-201`); QRF reported 15 distinct donor values. | Covered: monetary `match_reference`. Its pinned checkpoint fails at `d7b12bab` and passes the complete union repair. | +| `disability_benefits` | Nonnegative annual two-slot sum excluding workers' compensation (`disability_benefits.py:184-220,558-560`); QRF reported ten distinct donor values. | Inapplicable: `preserve_recipient` emits neither capacity nor selection evidence (`post_transfer_calibration.py:230-236,1319-1335`). Its preserved checkpoint validates with before/after carrier mass `42,658.57948297383`. | +| `weeks_unemployed` | Integer `-1` or `0..52`, with `-1` mapped to zero (`weeks_unemployed.py:791-800,911-983,1218-1222`); QRF reported 12 distinct donor values. | Covered: sole semantic count; positive-UC-constrained `match_reference` (`stacked_spine.py:8995-9008`). The exact replay proves reducer order, not count support, caused the failure. | +| `workers_compensation` | Exact nonnegative annual `WC_VAL` carry (`workers_compensation.py:143-184,520-522`); host had not reached it. | Covered: monetary default-mask `match_reference`; shared-kernel regressions validate its declaration. | +| `spm_unit_energy_subsidy` | Measured nonnegative annual `SPM_ENGVAL`, checked within unit and reduced to SPM-unit float64 (`energy_subsidy.py:157-233,543-557`); host had not reached it. | Covered: monetary default-mask `match_reference`; shared-kernel regressions validate its declaration at its entity grain. | + +The host log's near-discrete evidence appears at `build.log:1252-1266,1404-1408`. +Weeks does not rely on ACS's explicit discrete-numeric set, which contains only +two mortgage-year targets; ordinary numeric targets otherwise use the +continuous encoding. Its integer semantics come from the dedicated weeks +source/QRF checks and post-transfer donor-support mapping cited above +(`acs_transfer.py:129-138,3035-3117`). QRF's `<=32`-unique “near-discrete” +branch is a leaf-storage optimization, not a carrier-capacity semantic +distinction, which is why annual dollar targets also triggered it here +(`microcosm-fit/qrf.py:388-401,482-503`). + +Current zero-based late-DAG positions are child support 24, disability 25, +weeks 30, workers' compensation 31, energy subsidy 32, and adult care 34. +The registry constructs and schedules those groups deterministically, stacked +execution enumerates them serially, and each production group applies its +post-transfer calibration before returning +(`us_late_producer_registry.py:1338-1396,2013-2019`; +`stacked_spine.py:10054-10095,10927-10931`). The failed host run had crossed +child support and disability but had not produced checkpoints for workers' +compensation, energy subsidy, or adult care. Their verdict is therefore a +source/mask proof plus the six-spec shared-kernel regressions, not a claim of +checkpoint replay. + +## Source-cited mechanism record + +- The battery computes positive and negative carrier incidence separately, + compares ACS/ASEC weighted incidence to the frozen band, then computes five + weighted conditional carrier quantiles when both sides have enough rows + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:13351-13421`). +- Its quantile-envelope diagnostic is the maximum symmetric normalized + separation across those five conditional quantiles + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:13481-13512`). +- The fitted QRF draws a weighted sign gate and then a forest amount conditional + on the selected sign; that is the two-part mechanism calibrated here + (`packages/microcosm-fit/src/microcosm/fit/qrf.py:950-1003,1333-1429`). +- Ordinary ACS transfers partition recipients by optional-predictor + availability and construct exact complete-donor model frames. Regime + detection and fitted-model verification run only for an explicit + owner-selected target subset; the existing pattern seed and draw surface do + not change + (`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:919-1115,1297-1579`). +- The banked path applies the same explicit selection before the targetwise + chain and verifies only selected returned target regimes before accepting + their raw draws + (`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:1621-1848`). +- Adult-care qualification is a fail-closed section-21 predicate, and the + reconciliation clears nonqualifying mutable carriers, permits at most one + qualifying mutable carrier per tax unit, and preserves pre-existing positive + carriers + (`packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py:660-732`). +- The weeks-unemployed signal gate rejects positive PUF support without + unemployment compensation, so carrier additions must respect that compatible + capacity (`packages/microcosm-build/src/microcosm/build/us_runtime/weeks_unemployed.py:1328-1358`). +- The prior-year income gate explicitly requires the ASEC negative + self-employment leg to survive, so this lane may calibrate only the positive + mutable leg (`packages/microcosm-build/src/microcosm/build/us_runtime/prior_year_income.py:829-887`). + +These mechanisms support an artifact-side correction; they do not justify a +comparator change. + +## Implemented artifact mechanism + +- The immutable nine-target policy declares the exact early/late owner, carrier + mode, byte-exact negative leg, adult-care constraint, and weeks/UC constraint + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:123-263`). +- The kernel requires disjoint reference/recipient masks and mutable-subset + scope, snapshots and byte-compares protected surfaces, computes the reference + positive mass, and uses deterministic nearest-prefix removal/addition within + a proven attainable-mass interval + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:445-547,727-922`). +- Mutable positive amounts are mapped only to reference positive support and + are anchored at the frozen 10/25/50/75/90 percentiles; infeasible or + conflicting anchors are recorded rather than hidden + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:577-724`). +- The kernel proves boundary saturation when capacity-limited and rejects any + change to nonmutable, negative, negative-zero, or zero-weight bytes, any + donor-support escape, or any preserve-carrier change + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:939-978`). +- The stacked owner derives ASEC clone-0 reference rows, ACS clone-0 recipient + rows, and transferred-null mutable rows. Adult care uses qualifying rows plus + one candidate per empty unit; weeks uses positive-UC mutable rows. The final + adult-care reconciliation must be byte-identical/no-op + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8799-8955`). +- Schema-v2 receipts explicitly state that terminal validation cannot replay + pre-calibration state; they separate live-replayable output claims from + generation-transition evidence + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:43-119,980-1055`). +- Terminal validation independently recomputes live row/mask hashes, entity and + output hashes, full weights, carrier masses, reference/recipient quantiles, + QED, and coupled adult/weeks constraints + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:3756-3818,3821-4025`). +- Only the nine declared calibration targets receive QRF pattern receipts. + Those receipts persist ordered predictors, seeds, weights, row counts, and + selected regimes, then validate canonical predictor/pattern/target order and + exact record-family binding. All assigned and unassigned receipts validate + the same legacy row-count schema and accounting before their scope branch. + Unassigned transfers retain their evidence-free receipts and generic + serializers omit the empty opt-in field. Receipt-only validation deliberately + does not claim donor replay or out-of-sample verification + (`packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:4128-4445,4529-4795`). +- Canonical early and late production both use the certified maximum of eight + targets per fit. Exact selected-family receipt binding therefore cannot be + weakened by caller-selected batching; the non-production test seam retains + smaller-width coverage. +- The two pinned 3.73 GB SIPP readers retain chunked selection and downstream + explicit coercion while using streaming type inference. Guarded full-donor + reruns observed much lower RSS and unchanged locked donor facts + (`packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py:393-413`; + `packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py:375-397`). + +## Adjudication limits retained + +- The named adjudication remedy calls for target/sign-scoped origin-aware + cross-fitted carrier calibration and held-out nonregression. This branch does + not have the adjudicated fold/comparator authority in main and does not invent + one. Its carrier correction is deterministic terminal reference-margin + matching, not cross-fitting or an out-of-sample estimate + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:839-922`; + `packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:4211-4217`). +- The unemployment-compensation row likewise lacks the adjudicated money-OOS + authority in this branch. The implementation freezes its carrier membership + and calibrates only conditional positive amounts; no OOS nonregression claim + is made + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:193-207,952-978`). +- Weeks-unemployed carrier matching is allowed to stop at the exact + positive-UC-compatible capacity, but the receipt must prove the attainable + interval and boundary saturation + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:839-955`; + `packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:8863-8876`). +- Mutable masks, input hashes, before-state diagnostics, change counts, and + byte-preservation proofs need the generation-time pre-frame. Terminal + validation authenticates them through the enclosing execution authority and + does not claim to reconstruct them from the final frame + (`packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py:43-119`; + `packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py:3925-3938`). + +## Environment receipt + +- `uv sync --all-packages --extra us`: failed before resolution because the + sandbox denied writes to `/Users/maxghenis/.cache/uv`. +- Writable-cache retry: failed downloading `pandas==3.0.3` because sandbox DNS + is unavailable. +- Exact-lock fallback: `uv.lock` SHA-1 + `6b213e740b114d008c0191fa492832a957a0a948` matches + `../microcosm-707/uv.lock`; that environment imports NumPy 2.4.6, pandas + 3.0.3, and pytest 8.4.2 while `PYTHONPATH` points at this worktree. +- The pre-continuation PR test surface was green: all 225 `microcosm-build` + test files; `microcosm-fit` 93 passed; `microcosm-calibrate` 201 passed; + `microcosm-frame` 294 passed/36 skipped; and `microcosm-data` 275 passed/one + skipped. Heavy files ran in fresh pytest processes. The final continuation + tree also passes all 225 `microcosm-build` test files across fresh processes, + including full ordinary/banked transfer, stacked binding, serializer, pool, + H5 loader, and real late-executor files. The full rerun exposed one synthetic + H5 target receipt with only `residual_null_rows`; its fixture was updated to + the valid four-zero count block, its full file reran green, and a fixture + scan found no other canonical partial blocks. +- `ruff check .`, touched-file `ruff format --check`, and `git diff --check` + pass. Full-tree format checking reports 49 pre-existing files outside this + lane's formatting scope. +- On the completed post-transfer receipt repair, all 47 focused calibration + tests and all five package roots (`microcosm-fit`, `microcosm-calibrate`, + `microcosm-data`, `microcosm-frame`, and the complete `microcosm-build` + root) exited zero under the guard. Repository-wide Ruff, touched-file + formatting, and whitespace checks also pass. Only established skips and + warnings appeared; no host build ran. +- The final successful full-donor tests peaked at 0.485 GiB for vehicles and + 0.532 GiB for voluntary filing; the largest successful isolated build-test + shard peaked at 6.531 GiB. An earlier 13.5 GiB/250 ms diagnostic guard + observed one rapid parser spike at 15.424 GiB before termination. The guard + was immediately tightened to 10 GiB/20 ms, the streaming reader was fixed, + and all successful reruns stayed below the cap. + +## Measurement ledger + +The before build used `--sample-fraction 0.01 --sample-seed 578`, clone fraction +1 and clone seed 578, with no chain predecessor. Peak observed per-process RSS +was 10.733 GiB. Artifact receipts: + +- `pool.h5`: SHA-256 `258891504201275f8006a1584b7d3e891890d15381724bc9f2b30b1f443d967f` +- `pool.gates.json`: SHA-256 `94ee914bb7490e7f513184e691cf15847d1e585693ef184977196485f86f1fee` +- `pool.manifest.json`: SHA-256 `953aaff72fac8cc17211959d0133f9bce8c22dd87b2957a0f6a89335fcc9c122` + +The exact frozen-battery values are below. `unsupported` means the battery did +not emit QED because one side had fewer than five carriers; the parenthesized +number is the same five-grid weighted diagnostic computed manually without +changing that support rule. + +| Assigned check | Before at 1% | +| --- | ---: | +| adult care positive incidence ratio | 0.561425035 | +| adult care positive QED | unsupported (manual 1.738865343) | +| unemployment compensation positive QED | 0.352941176 | +| child-support expense positive incidence ratio | 0.171280844 | +| child-support expense positive QED | 0.953846154 | +| child support received positive incidence ratio | 0.242882414 | +| child support received positive QED | 1.000000000 | +| disability benefits positive QED | 1.373534621 | +| prior-year self-employment positive incidence ratio | 1.376752468 | +| prior-year self-employment positive QED | 0.834720589 | +| weeks unemployed positive incidence ratio | 0.025384419 | +| weeks unemployed positive QED | 0.736842105 | +| workers' compensation positive incidence ratio | 0.047614655 | +| workers' compensation positive QED | unsupported (manual 1.918367347) | +| SPM-unit energy subsidy positive incidence ratio | 0.240477284 | +| SPM-unit energy subsidy positive QED | 0.666666667 | + +Sparse 1% sampling also puts the otherwise frozen unemployment-compensation +and disability positive carrier ratios at 0.131859569 and 0.098471480. Their +adjudicated remedies freeze carrier membership, so this lane will improve only +their assigned amount-QED checks. The unassigned negative prior-year +self-employment ratio is 1.435953092 and is likewise deliberately untouched. + +After values will be added after the calibrated 1% rebuild. +## Historical lane notes imported from `origin/main` — 2026-08-22 + +The following `one-target-surface` notes are preserved verbatim as merge +history. Any present-tense branch state or pending work below belongs to that +source lane's 2026-08-21 snapshot, not to this continuation. + # One-target-surface lane notes ## 2026-08-21 — baseline and doctrine diff --git a/changelog.d/742-us-post-transfer-calibration-policy-spec.added.md b/changelog.d/742-us-post-transfer-calibration-policy-spec.added.md new file mode 100644 index 00000000..7e165d4a --- /dev/null +++ b/changelog.d/742-us-post-transfer-calibration-policy-spec.added.md @@ -0,0 +1 @@ +Declare the US post-transfer calibration policy in the authored F0 imputation spec as a closed, typed exact variant of the regime-gated QRF model, projected as a stacked-authority component and held byte-equal to the runtime policy identity. diff --git a/docs/evidence/spec-engine/us-f0-coverage.json b/docs/evidence/spec-engine/us-f0-coverage.json index 482879d4..6dd5d19c 100644 --- a/docs/evidence/spec-engine/us-f0-coverage.json +++ b/docs/evidence/spec-engine/us-f0-coverage.json @@ -6,7 +6,7 @@ "country": "us", "documentation_sha256": "aed5b95645a5c605671e374ed2a9d511c7fd9dc29c63760c32a205d9f2db9491", "field_usage": { - "authored_normative_field_count": 32160, + "authored_normative_field_count": 32252, "claim_count": 47, "claims": [ { @@ -186,8 +186,8 @@ ], "mode": "legacy_behavior", "pointer_class": "all", - "pointer_count": 4, - "pointer_sha256": "93d4b493904c7318725feb66dc594bba4eef7cdd73fc983397572414d45ce6a8", + "pointer_count": 96, + "pointer_sha256": "e4d6b6b747fcec1c027e0f1c2d1905274c0426217a61383b02e75baadb93db4d", "rationale": null, "relative_sink_prefix": null, "source_prefix": "/authored/spec~1imputation.yaml/models", @@ -740,20 +740,20 @@ "verifier": "vintages" } ], - "configuration_field_count": 41379, - "consumed_field_count": 41379, + "configuration_field_count": 41471, + "consumed_field_count": 41471, "generation0_effect_counts": { - "legacy_behavior": 37717, + "legacy_behavior": 37809, "no_generation0_effect": 3662 }, "mode_counts": { "compiler_semantic": 27661, "front_end_validation": 346, "identity_only": 128, - "legacy_behavior": 13244 + "legacy_behavior": 13336 }, "multiple_primary_use_field_count": 0, - "pointer_inventory_sha256": "b9d2932e0ae5efb6637be7b5d90e33b2e159ffb807ee50ff36ee2b51296cfdc4", + "pointer_inventory_sha256": "2daa3ee07ac2e5d5ab731348edbca8c7a58438e9819d1ac3707070c7500a1c63", "resolved_binding_field_count": 9219, "unused_field_count": 0 }, @@ -765,7 +765,7 @@ "country": "us", "counts": { "adapter_surfaces": 13, - "authority_components": 8, + "authority_components": 9, "early_families": 13, "early_targets": 48, "itemization_batches": 5, @@ -1624,13 +1624,13 @@ "compiler_ir.node_slices" ], "expected": { - "map_sha256": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "protocol_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec" + "map_sha256": "cf5037426fda1c258c40ae8e6cc6381e27bf094eb1063e6dd40b901e1554512a", + "protocol_sha256": "5d24161d335ac39d63d904aab4ddd1157be5601a62cdd9523797e7466e784535" }, "failures": [], "observed": { - "map_sha256": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "protocol_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec" + "map_sha256": "cf5037426fda1c258c40ae8e6cc6381e27bf094eb1063e6dd40b901e1554512a", + "protocol_sha256": "5d24161d335ac39d63d904aab4ddd1157be5601a62cdd9523797e7466e784535" }, "status": "covered" }, @@ -1645,7 +1645,7 @@ "compiler_ir.seed_stream_map" ], "expected": { - "implementation_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "implementation_sha256": "5d24161d335ac39d63d904aab4ddd1157be5601a62cdd9523797e7466e784535", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1666,7 +1666,7 @@ }, "failures": [], "observed": { - "implementation_sha256": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "implementation_sha256": "5d24161d335ac39d63d904aab4ddd1157be5601a62cdd9523797e7466e784535", "protocol": "legacy-v1", "streams": [ "build_model", @@ -1776,7 +1776,8 @@ "joint_metric_registry", "support_profile", "puf_capital_gains_tail_support_contract", - "late_producer_schedule" + "late_producer_schedule", + "post_transfer_calibration" ] }, "failures": [], @@ -1789,7 +1790,8 @@ "joint_metric_registry", "support_profile", "puf_capital_gains_tail_support_contract", - "late_producer_schedule" + "late_producer_schedule", + "post_transfer_calibration" ] }, "status": "covered" @@ -1806,11 +1808,11 @@ "legacy_adapter.stacked_checkpoint_static_components" ], "expected": { - "sha256": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "sha256": "24ae81f408e8d608db1956463206fe9842b5214ca813113d3e5366fc66ca3cde" }, "failures": [], "observed": { - "sha256": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "sha256": "24ae81f408e8d608db1956463206fe9842b5214ca813113d3e5366fc66ca3cde" }, "status": "covered" }, @@ -1852,7 +1854,7 @@ "alpha", "zeta" ], - "sha256": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8" + "sha256": "09cb19f84333a5eeabc4b5adeeef0632050bc0f7810a4ba14a6727c11d681377" }, "failures": [], "observed": { @@ -1875,7 +1877,7 @@ "alpha", "zeta" ], - "sha256": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8" + "sha256": "09cb19f84333a5eeabc4b5adeeef0632050bc0f7810a4ba14a6727c11d681377" }, "status": "covered" }, @@ -2438,7 +2440,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "586491f0866180f7a8f1e01530af5c2a3f2ebc18ad289ac1a3385d1897e79626" + "spec_sha256": "d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0" } }, "report_schema_version": 3, @@ -2448,7 +2450,7 @@ "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "586491f0866180f7a8f1e01530af5c2a3f2ebc18ad289ac1a3385d1897e79626" + "spec_sha256": "d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0" }, "status": "pass" } diff --git a/experiments/battery_burndown_pkg3_evidence.md b/experiments/battery_burndown_pkg3_evidence.md new file mode 100644 index 00000000..03bdb704 --- /dev/null +++ b/experiments/battery_burndown_pkg3_evidence.md @@ -0,0 +1,29 @@ +# pkg3 1% verify build: battery failure-line diff vs baseline + +Same code path, sample fraction 0.01 seed 578, checkpoint-resumed; +physical (mirror-deduplicated) failure lines from pool.gates.json. + +- baseline1pct: **127** failures +- pkg3: **114** failures +- greened: **15**, introduced: **2** + +## Greened by pkg3 +- person/adult_care/pre_subsidy_care_expenses[clone_0]/positive: weighted positive-leg incidence ratio 0.561425 is outside [0.8, 1.25] (asec=0.000768207, acs=0.000431291). +- person/model_required_numeric/unemployment_compensation[clone_0]/positive: conditional-quantile envelope distance 0.352941 exceeds 0.25. +- person/simulated_output/ssi[clone_0]/positive: weighted positive-leg incidence ratio 1.34148 is outside [0.8, 1.25] (asec=0.0203726, acs=0.0273295). +- person/source_operator_child_support/child_support_expense[clone_0]/positive: conditional-quantile envelope distance 0.953846 exceeds 0.25. +- person/source_operator_child_support/child_support_expense[clone_0]/positive: weighted positive-leg incidence ratio 0.171281 is outside [0.8, 1.25] (asec=0.00523716, acs=0.000897025). +- person/source_operator_child_support/child_support_received[clone_0]/positive: conditional-quantile envelope distance 1 exceeds 0.25. +- person/source_operator_child_support/child_support_received[clone_0]/positive: weighted positive-leg incidence ratio 0.242882 is outside [0.8, 1.25] (asec=0.00928306, acs=0.00225469). +- person/source_operator_disability_benefits/disability_benefits[clone_0]/positive: conditional-quantile envelope distance 1.37353 exceeds 0.25. +- person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/positive: conditional-quantile envelope distance 0.834721 exceeds 0.25. +- person/source_operator_prior_year_income/self_employment_income_last_year[clone_0]/positive: weighted positive-leg incidence ratio 1.37675 is outside [0.8, 1.25] (asec=0.0249193, acs=0.0343077). +- person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive: conditional-quantile envelope distance 0.736842 exceeds 0.25. +- person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive: weighted positive-leg incidence ratio 0.0253844 is outside [0.8, 1.25] (asec=0.0341695, acs=0.000867374). +- person/source_operator_workers_compensation/workers_compensation[clone_0]/positive: weighted positive-leg incidence ratio 0.0476147 is outside [0.8, 1.25] (asec=0.00326463, acs=0.000155444). +- spm_unit/source_operator_energy_subsidy/spm_unit_energy_subsidy[clone_0]/positive: conditional-quantile envelope distance 0.666667 exceeds 0.25. +- spm_unit/source_operator_energy_subsidy/spm_unit_energy_subsidy[clone_0]/positive: weighted positive-leg incidence ratio 0.240477 is outside [0.8, 1.25] (asec=0.033554, acs=0.00806897). + +## Present only under pkg3 (same checks, ratios moved toward band, still red) +- person/simulated_output/ssi[clone_0]/positive: weighted positive-leg incidence ratio 1.33548 is outside [0.8, 1.25] (asec=0.0203726, acs=0.0272073). +- person/source_operator_weeks_unemployed/weeks_unemployed[clone_0]/positive: weighted positive-leg incidence ratio 0.0313711 is outside [0.8, 1.25] (asec=0.0341695, acs=0.00107194). diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py index fb7a04c0..21bf1806 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/field_usage.py @@ -26,9 +26,9 @@ ) from .schemas import load_schema_registry -EXPECTED_AUTHORED_FIELD_COUNT = 32_160 +EXPECTED_AUTHORED_FIELD_COUNT = 32_252 EXPECTED_RESOLVED_BINDING_FIELD_COUNT = 9_219 -EXPECTED_CONFIGURATION_FIELD_COUNT = 41_379 +EXPECTED_CONFIGURATION_FIELD_COUNT = 41_471 class FieldUsageError(AssertionError): @@ -357,8 +357,8 @@ def _path_inventory(rows: Sequence[tuple[str, object]]) -> tuple[int, str]: "deb4acc6f962df9eb458e4206e1a75a3fe596a9a9cdb9c9163e75fd8d4f7b77a", ), "imputation_models": ( - 4, - "93d4b493904c7318725feb66dc594bba4eef7cdd73fc983397572414d45ce6a8", + 96, + "e4d6b6b747fcec1c027e0f1c2d1905274c0426217a61383b02e75baadb93db4d", ), "imputation_predictor_blocks": ( 62, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py index 9bf9ea50..cf446334 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/inventory_coverage.py @@ -47,6 +47,7 @@ "support_profile", "puf_capital_gains_tail_support_contract", "late_producer_schedule", + "post_transfer_calibration", ) EXPECTED_CHECKPOINT_TOP_LEVEL = frozenset( @@ -330,9 +331,9 @@ EXPECTED_HASHES = { "acs_group_predictors": "a927bb7ecf3e84f54c93583ab79318654514ac546aefafba67da5285615fbd60", "acs_person_predictors": "878c788a6f037d7aca12b3586ea034eff04f3034ffa11935a736493042551f25", - "authority": "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d", + "authority": "24ae81f408e8d608db1956463206fe9842b5214ca813113d3e5366fc66ca3cde", "early_families": "e26a90e2b5c16e23e7c17424d1c2e4ab18ed66b1d0e129248e87c0bab9b3fd5d", - "full_checkpoint": "2972db18e0f69ac4df4079916355a1bcd9ec5c12f3e19d5650fc47b5a8a0e3e8", + "full_checkpoint": "09cb19f84333a5eeabc4b5adeeef0632050bc0f7810a4ba14a6727c11d681377", "gap_fill_schedule": "96aefe2853de91ae95f50bc2ccc2c1dd94802c27f21c643981152bbcb13c4e10", "graph_nodes": "a83363de26cad0144b5a98b36b4bca49542e37a7b9fee3d7e541f692deeff864", "late_families": "a160432fc12a85df20ba7fd6687673b3c31786df7a983e2477604ab923b26d18", @@ -340,8 +341,8 @@ "late_schedule": "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5", "ownership": "5f64f0aac49e2313177564f71876bffc8c81b3ded4df701e70930e60e9c98356", "primary_tuples": "987b501c695e31f45521c4a178528f75ab3df22c09bc407b182213b2de99ee57", - "seed_map": "96140220b6b248c1b3a3567dc0c97df6c08176e6745d8dd55786053f26c43a32", - "seed_protocol": "6dade07562ec29c56d96ab8e299a4416c679f1c44b18b228e0ef10f21bd6f6ec", + "seed_map": "cf5037426fda1c258c40ae8e6cc6381e27bf094eb1063e6dd40b901e1554512a", + "seed_protocol": "5d24161d335ac39d63d904aab4ddd1157be5601a62cdd9523797e7466e784535", "source_manifest": "16f64b9cf3aea326737accca64c742a1edaa30a3a499432efd7567942f38a6c7", "take_up": "495dc6ed195eae372a6ba098c6fb894323638a4a7dce1b4fe7efaaf6beb69446", "tail": "ac92829c88a1a4fb6460d61190918d5d99c6c377fc8dd8f62f02b332d09bf59c", @@ -396,7 +397,7 @@ EXPECTED_INVENTORY_COUNTS: Mapping[str, int] = { "adapter_surfaces": 13, - "authority_components": 8, + "authority_components": 9, "early_families": 13, "early_targets": 48, "itemization_batches": 5, @@ -470,8 +471,7 @@ def _without_operational_bindings(value: object) -> object: return { key: _without_operational_bindings(item) for key, item in value.items() - if key != "worker_execution" - and not (drop_self_hash and key == "sha256") + if key != "worker_execution" and not (drop_self_hash and key == "sha256") } if isinstance(value, list): return [_without_operational_bindings(item) for item in value] @@ -1520,9 +1520,9 @@ def add( full_resource_semantics.get("producers"), "full checkpoint resource-semantics producers", ) - if _mapping( - value, "full checkpoint resource-semantics producer" - ).get("producer") + if _mapping(value, "full checkpoint resource-semantics producer").get( + "producer" + ) == "primary_puf_qrf" ] primary_resource_row = ( @@ -1558,9 +1558,7 @@ def add( "alpha": {"sha256": "a" * 64, "size_bytes": 17}, "zeta": {"sha256": "b" * 64, "size_bytes": 23}, }, - "full checkpoint sampling vector differs": full_checkpoint.get( - "sampling" - ) + "full checkpoint sampling vector differs": full_checkpoint.get("sampling") == { "sample_fraction": 0.25, "fraction_token": "f025", @@ -1592,9 +1590,7 @@ def add( observed={ "sha256": _operational_free_sha256(full_checkpoint), "field_names": sorted(full_checkpoint), - "input_roles": list( - _mapping(full_checkpoint["inputs"], "full inputs") - ), + "input_roles": list(_mapping(full_checkpoint["inputs"], "full inputs")), "fraction_token": _mapping( full_checkpoint["sampling"], "full sampling" ).get("fraction_token"), @@ -1963,8 +1959,10 @@ def assert_inventory_coverage_complete(report: Mapping[str, object]) -> None: ): failures.append("inventory spec_binding contract differs") spec_sha256 = binding.get("spec_sha256") - if not isinstance(spec_sha256, str) or len(spec_sha256) != 64 or any( - character not in "0123456789abcdef" for character in spec_sha256 + if ( + not isinstance(spec_sha256, str) + or len(spec_sha256) != 64 + or any(character not in "0123456789abcdef" for character in spec_sha256) ): failures.append("inventory spec_binding SHA-256 is invalid") @@ -1972,13 +1970,16 @@ def assert_inventory_coverage_complete(report: Mapping[str, object]) -> None: if not isinstance(compiler_abi, Mapping): failures.append("inventory compiler_ir_abi is missing") else: - if set(compiler_abi) != {"version", "sha256"} or compiler_abi.get( - "version" - ) != 1: + if ( + set(compiler_abi) != {"version", "sha256"} + or compiler_abi.get("version") != 1 + ): failures.append("inventory compiler IR ABI contract differs") abi_sha256 = compiler_abi.get("sha256") - if not isinstance(abi_sha256, str) or len(abi_sha256) != 64 or any( - character not in "0123456789abcdef" for character in abi_sha256 + if ( + not isinstance(abi_sha256, str) + or len(abi_sha256) != 64 + or any(character not in "0123456789abcdef" for character in abi_sha256) ): failures.append("inventory compiler IR ABI SHA-256 is invalid") @@ -2028,8 +2029,7 @@ def assert_inventory_coverage_complete(report: Mapping[str, object]) -> None: not isinstance(homes, list) or not homes or any( - not isinstance(home, str) or not home.startswith("/") - for home in homes + not isinstance(home, str) or not home.startswith("/") for home in homes ) ): failures.append(f"{name}: bundle_homes are malformed or empty") @@ -2059,8 +2059,7 @@ def assert_inventory_coverage_complete(report: Mapping[str, object]) -> None: recomputed_covered = recomputed_required - len(recomputed_missing) if recomputed_missing: failures.append( - "inventory has missing required items: " - + ", ".join(recomputed_missing) + "inventory has missing required items: " + ", ".join(recomputed_missing) ) expected_summaries = { "required_item_count": recomputed_required, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json index e0e87919..0c2a2548 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/schema/imputation.schema.json @@ -65,43 +65,11 @@ "$ref": "defs.schema.json#/$defs/identifier" }, "additionalProperties": { - "type": "object", - "additionalProperties": false, - "required": [ - "kernel", - "params" - ], - "properties": { - "kernel": { - "$ref": "defs.schema.json#/$defs/kernel_ref" - }, - "params": { - "type": "object", - "additionalProperties": false, - "required": [ - "n_estimators", - "max_samples_leaf", - "zero_atol" - ], - "properties": { - "n_estimators": { - "type": "integer", - "minimum": 1 - }, - "max_samples_leaf": { - "type": [ - "integer", - "null" - ], - "minimum": 1 - }, - "zero_atol": { - "type": "number", - "exclusiveMinimum": 0 - } - } + "oneOf": [ + { + "$ref": "#/$defs/regime_gated_qrf_model" } - } + ] } }, "chaining": { @@ -775,6 +743,260 @@ } }, "$defs": { + "regime_gated_qrf_model": { + "type": "object", + "additionalProperties": false, + "required": [ + "kernel", + "params", + "post_draw_calibration" + ], + "properties": { + "kernel": { + "const": "kernel:regime_gated_qrf" + }, + "params": { + "type": "object", + "additionalProperties": false, + "required": [ + "n_estimators", + "max_samples_leaf", + "zero_atol" + ], + "properties": { + "n_estimators": { + "type": "integer", + "minimum": 1 + }, + "max_samples_leaf": { + "type": [ + "integer", + "null" + ], + "minimum": 1 + }, + "zero_atol": { + "type": "number", + "exclusiveMinimum": 0 + } + } + }, + "post_draw_calibration": { + "$ref": "#/$defs/post_draw_calibration_policy" + } + } + }, + "post_draw_calibration_policy": { + "oneOf": [ + { + "$ref": "#/$defs/post_transfer_calibration_policy_v1" + } + ] + }, + "post_transfer_calibration_policy_v1": { + "type": "object", + "additionalProperties": false, + "required": [ + "artifact_kind", + "schema_version", + "scope", + "quantiles", + "carrier_selection", + "amount_mapping", + "targets", + "sha256" + ], + "properties": { + "artifact_kind": { + "const": "microcosm_us_post_transfer_calibration_policy" + }, + "schema_version": { + "const": 1 + }, + "scope": { + "type": "object", + "additionalProperties": false, + "required": [ + "reference", + "recipient", + "mutable", + "provenance_masks", + "constraint_masks", + "value_dtype", + "zero_weight_rows" + ], + "properties": { + "reference": { + "type": "string", + "minLength": 1 + }, + "recipient": { + "type": "string", + "minLength": 1 + }, + "mutable": { + "type": "string", + "minLength": 1 + }, + "provenance_masks": { + "type": "string", + "minLength": 1 + }, + "constraint_masks": { + "type": "string", + "minLength": 1 + }, + "value_dtype": { + "type": "string", + "minLength": 1 + }, + "zero_weight_rows": { + "type": "string", + "minLength": 1 + } + } + }, + "quantiles": { + "type": "array", + "items": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "minItems": 1, + "uniqueItems": true + }, + "carrier_selection": { + "type": "object", + "additionalProperties": false, + "required": [ + "match_reference", + "removal_order", + "addition_order", + "equal_distance" + ], + "properties": { + "match_reference": { + "type": "string", + "minLength": 1 + }, + "removal_order": { + "type": "string", + "minLength": 1 + }, + "addition_order": { + "type": "string", + "minLength": 1 + }, + "equal_distance": { + "type": "string", + "minLength": 1 + } + } + }, + "amount_mapping": { + "type": "object", + "additionalProperties": false, + "required": [ + "leg", + "recipient_rank", + "inverse_cdf", + "exact_quantile_anchors", + "infeasible_anchor_handling", + "output_support" + ], + "properties": { + "leg": { + "type": "string", + "minLength": 1 + }, + "recipient_rank": { + "type": "string", + "minLength": 1 + }, + "inverse_cdf": { + "type": "string", + "minLength": 1 + }, + "exact_quantile_anchors": { + "type": "array", + "items": { + "type": "number", + "minimum": 0, + "maximum": 1 + }, + "minItems": 1, + "uniqueItems": true + }, + "infeasible_anchor_handling": { + "type": "string", + "minLength": 1 + }, + "output_support": { + "type": "string", + "minLength": 1 + } + } + }, + "targets": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "entity", + "family", + "target", + "stage", + "carrier_mode", + "negative_leg", + "special_constraint" + ], + "properties": { + "entity": { + "type": "string", + "minLength": 1 + }, + "family": { + "type": "string", + "minLength": 1 + }, + "target": { + "type": "string", + "minLength": 1 + }, + "stage": { + "enum": [ + "early_gap_fill", + "late_transfer" + ] + }, + "carrier_mode": { + "enum": [ + "match_reference", + "preserve_recipient" + ] + }, + "negative_leg": { + "const": "byte_exact" + }, + "special_constraint": { + "enum": [ + "none", + "adult_care_qualifying_one_per_tax_unit", + "weeks_requires_positive_unemployment_compensation" + ] + } + } + }, + "minItems": 1, + "uniqueItems": true + }, + "sha256": { + "$ref": "defs.schema.json#/$defs/sha256" + } + } + }, "node_capabilities": { "type": "object", "additionalProperties": false, diff --git a/packages/microcosm-build/src/microcosm/build/spec_engine/stacked_authority_semantics.py b/packages/microcosm-build/src/microcosm/build/spec_engine/stacked_authority_semantics.py index 74e65e2a..2c26a666 100644 --- a/packages/microcosm-build/src/microcosm/build/spec_engine/stacked_authority_semantics.py +++ b/packages/microcosm-build/src/microcosm/build/spec_engine/stacked_authority_semantics.py @@ -146,9 +146,7 @@ def _late_families( for index, value in enumerate( _array(imputation.get("families"), location="imputation/families") ) - for family in [ - _mapping(value, location=f"imputation/families/{index}") - ] + for family in [_mapping(value, location=f"imputation/families/{index}")] if family.get("stage") == "late_producer_dag" ) @@ -183,9 +181,7 @@ def _late_transfer_surface( ): target = _mapping( value, - location=( - f"imputation/late_families/{index}/targets/{target_index}" - ), + location=(f"imputation/late_families/{index}/targets/{target_index}"), ) name = _nonempty_string( target.get("name"), @@ -208,9 +204,7 @@ def _late_transfer_surface( ) projection = _mapping( donor.get("projection"), - location=( - f"imputation/late_families/{index}/donor_contract/projection" - ), + location=(f"imputation/late_families/{index}/donor_contract/projection"), ) donor_channels.add( _nonempty_string( @@ -239,8 +233,7 @@ def _late_transfer_surface( _nonempty_string( recipient.get("selection"), location=( - "imputation/late_families/" - f"{index}/recipient_contract/selection" + f"imputation/late_families/{index}/recipient_contract/selection" ), ) ) @@ -277,9 +270,7 @@ def _primary_puf_output_keys( ) -> set[tuple[str, str]]: primary = [ family - for value in _array( - imputation.get("families"), location="imputation/families" - ) + for value in _array(imputation.get("families"), location="imputation/families") for family in [_mapping(value, location="imputation/families row")] if family.get("stage") == "primary_puf_qrf" ] @@ -412,6 +403,22 @@ def _authority_component_payloads( imputation_payloads: Mapping[str, object], ) -> tuple[dict[str, object], dict[str, object]]: battery = project_battery_authority_components(domains["battery"]) + models = _mapping( + domains["imputation"].get("models"), + location="imputation/models", + ) + regime_gated_qrf = _mapping( + models.get("regime_gated_qrf"), + location="imputation/models/regime_gated_qrf", + ) + post_transfer_calibration = deepcopy( + dict( + _mapping( + regime_gated_qrf.get("post_draw_calibration"), + location=("imputation/models/regime_gated_qrf/post_draw_calibration"), + ) + ) + ) transfer_surface, donor_channel, donor_clone_index, recipient_selection = ( _late_transfer_surface(domains["imputation"]) ) @@ -456,6 +463,7 @@ def _authority_component_payloads( "support_profile": deepcopy(battery["support_profile"]), "puf_capital_gains_tail_support_contract": tail_support, "late_producer_schedule": late_schedule, + "post_transfer_calibration": post_transfer_calibration, } counts = { "gap_fill_target_count": _gap_fill_target_count(gap_fill_plan), @@ -487,6 +495,14 @@ def _authority_component_payloads( location="battery projection/joint_metric_registry", ) ), + "post_transfer_calibration_target_count": len( + _array( + post_transfer_calibration.get("targets"), + location=( + "imputation/models/regime_gated_qrf/post_draw_calibration/targets" + ), + ) + ), } return payloads, counts @@ -545,9 +561,7 @@ def project_stacked_authority_receipt(spec: ResolvedSpec) -> dict[str, object]: "declared_sha256": digests["post_puf_transfer_surface"], "target_count": counts["post_puf_target_count"], "puf_producer_target_count": counts["puf_producer_target_count"], - "source_producer_target_count": counts[ - "source_producer_target_count" - ], + "source_producer_target_count": counts["source_producer_target_count"], "donor_channel": _mapping( payloads["post_puf_transfer_surface"], location="authority/post_puf_transfer_surface", @@ -588,13 +602,9 @@ def project_stacked_authority_receipt(spec: ResolvedSpec) -> dict[str, object]: "digest_matches_declared": True, }, "puf_capital_gains_tail_support_contract": { - "identity": deepcopy( - payloads["puf_capital_gains_tail_support_contract"] - ), + "identity": deepcopy(payloads["puf_capital_gains_tail_support_contract"]), "sha256": digests["puf_capital_gains_tail_support_contract"], - "declared_sha256": digests[ - "puf_capital_gains_tail_support_contract" - ], + "declared_sha256": digests["puf_capital_gains_tail_support_contract"], "digest_matches_declared": True, }, "late_producer_schedule": { @@ -605,6 +615,13 @@ def project_stacked_authority_receipt(spec: ResolvedSpec) -> dict[str, object]: "producer_count": late_schedule.get("producer_count"), "digest_matches_declared": True, }, + "post_transfer_calibration": { + "identity": deepcopy(payloads["post_transfer_calibration"]), + "sha256": digests["post_transfer_calibration"], + "declared_sha256": digests["post_transfer_calibration"], + "target_count": counts["post_transfer_calibration_target_count"], + "digest_matches_declared": True, + }, } return { "authority_id": authority_id, @@ -658,15 +675,12 @@ def _tail_support_contract( ) contract = _mapping( tail.get("legacy_contract"), - location=( - "spine/support_roles/puf_tax_detail/tail_support/legacy_contract" - ), + location=("spine/support_roles/puf_tax_detail/tail_support/legacy_contract"), ) schema_version = _integer( tail.get("manifest_schema_version"), location=( - "spine/support_roles/puf_tax_detail/tail_support/" - "manifest_schema_version" + "spine/support_roles/puf_tax_detail/tail_support/manifest_schema_version" ), minimum=1, ) @@ -747,9 +761,7 @@ def _take_up_identity( asserted = _mapping( legacy.get("asserted_engine"), location="projected take_up/asserted_engine" ) - programs = _array( - legacy.get("programs"), location="projected take_up/programs" - ) + programs = _array(legacy.get("programs"), location="projected take_up/programs") return { "version": legacy["version"], "country": legacy["country"], @@ -808,13 +820,14 @@ def _rung_token(publication: Mapping[str, object], fraction: float) -> str: location="publication/release/rung_fractions", ) ): - row = _mapping( - value, location=f"publication/release/rung_fractions/{index}" - ) - if _finite_float( - row.get("fraction"), - location=f"publication/release/rung_fractions/{index}/fraction", - ) == fraction: + row = _mapping(value, location=f"publication/release/rung_fractions/{index}") + if ( + _finite_float( + row.get("fraction"), + location=f"publication/release/rung_fractions/{index}/fraction", + ) + == fraction + ): matches.append( _nonempty_string( row.get("token"), @@ -869,9 +882,7 @@ def _resolved_late_resource_semantics( matches = [ _mapping(value, location="late resource semantics producer") for value in producers - if _mapping( - value, location="late resource semantics producer" - ).get("producer") + if _mapping(value, location="late resource semantics producer").get("producer") == "primary_puf_qrf" ] if len(matches) != 1: @@ -924,9 +935,7 @@ def _model_parameters(imputation: Mapping[str, object]) -> tuple[int, int]: location="imputation/families/max_targets_per_fit", minimum=1, ) - for value in _array( - imputation.get("families"), location="imputation/families" - ) + for value in _array(imputation.get("families"), location="imputation/families") for family in [_mapping(value, location="imputation/families row")] if family.get("max_targets_per_fit") is not None } @@ -1044,8 +1053,7 @@ def project_stacked_checkpoint_base_identity( _array( pipeline.get("pre_clone_source_operator_order"), location=( - "spine/pipeline_contract/" - "pre_clone_source_operator_order" + "spine/pipeline_contract/pre_clone_source_operator_order" ), ) ) @@ -1058,8 +1066,7 @@ def project_stacked_checkpoint_base_identity( _array( pipeline.get("post_clone_source_operator_order"), location=( - "spine/pipeline_contract/" - "post_clone_source_operator_order" + "spine/pipeline_contract/post_clone_source_operator_order" ), ) ) @@ -1116,8 +1123,7 @@ def project_stacked_checkpoint_base_identity( "simulation_household_batch_size": _integer( simulation_batch.get("value"), location=( - "spine/pipeline_contract/" - "simulation_household_batch_size/value" + "spine/pipeline_contract/simulation_household_batch_size/value" ), minimum=1, ), @@ -1174,9 +1180,7 @@ def project_stacked_checkpoint_static_components( attachment.get("fraction"), location="spine/support_roles/puf_tax_detail/attachment/fraction", ).get("default"), - location=( - "spine/support_roles/puf_tax_detail/attachment/fraction/default" - ), + location=("spine/support_roles/puf_tax_detail/attachment/fraction/default"), ) attachment_seed = _integer( _mapping( diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/battery.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/battery.yaml index aed9ddc9..63fae35e 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/battery.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/battery.yaml @@ -823,4 +823,4 @@ metric_contracts: threshold: categorical_total_variation authority_binding: authority_id: us_stacked_spine_authority - version: 10 + version: 11 diff --git a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml index e7bd93c4..f42e8136 100644 --- a/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml +++ b/packages/microcosm-build/src/microcosm/build/us/spec/imputation.yaml @@ -91,6 +91,105 @@ models: n_estimators: 100 max_samples_leaf: null zero_atol: 1.0e-06 + post_draw_calibration: + artifact_kind: microcosm_us_post_transfer_calibration_policy + schema_version: 1 + scope: + reference: asec_origin_clone_0 + recipient: acs_origin_clone_0 + mutable: caller_supplied_target_cells + provenance_masks: caller_supplied_no_internal_inference + constraint_masks: caller_supplied_hash_bound + value_dtype: float64_byte_contract + zero_weight_rows: byte_exact + quantiles: + - 0.1 + - 0.25 + - 0.5 + - 0.75 + - 0.9 + carrier_selection: + match_reference: weighted_positive_prevalence_nearest_prefix + removal_order: positive_amount_descending_then_entity_id + addition_order: entity_id + equal_distance: lower_mass + amount_mapping: + leg: positive + recipient_rank: weighted_full_recipient_positive_upper_cdf + inverse_cdf: left + exact_quantile_anchors: + - 0.1 + - 0.25 + - 0.5 + - 0.75 + - 0.9 + infeasible_anchor_handling: frame_owner_fail_closed + output_support: reference_positive_values_only + targets: + - entity: person + family: adult_care + target: pre_subsidy_care_expenses + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: adult_care_qualifying_one_per_tax_unit + - entity: person + family: model_required_numeric + target: unemployment_compensation + stage: early_gap_fill + carrier_mode: preserve_recipient + negative_leg: byte_exact + special_constraint: none + - entity: person + family: source_operator_child_support + target: child_support_expense + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: none + - entity: person + family: source_operator_child_support + target: child_support_received + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: none + - entity: person + family: source_operator_disability_benefits + target: disability_benefits + stage: late_transfer + carrier_mode: preserve_recipient + negative_leg: byte_exact + special_constraint: none + - entity: person + family: source_operator_prior_year_income + target: self_employment_income_last_year + stage: early_gap_fill + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: none + - entity: person + family: source_operator_weeks_unemployed + target: weeks_unemployed + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: weeks_requires_positive_unemployment_compensation + - entity: person + family: source_operator_workers_compensation + target: workers_compensation + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: none + - entity: spm_unit + family: source_operator_energy_subsidy + target: spm_unit_energy_subsidy + stage: late_transfer + carrier_mode: match_reference + negative_leg: byte_exact + special_constraint: none + sha256: ff609afd95d8a1443a5c32b08dc9d444c439d802a77972158c355afc56955332 chaining: order: declared splits: declared_only diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_multispine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_multispine.py index de0cc308..2cf781f0 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_multispine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_multispine.py @@ -295,6 +295,11 @@ def _json_ready(value: Any) -> Any: "ACS multispine provenance mapping keys must be strings; " f"got {type(key).__name__}." ) + if key == "target_regimes" and not item: + # The opt-in stacked audit field did not exist on legacy ACS + # multispine pattern provenance. Keep default callers' JSON + # schema byte-compatible when the audit selection is empty. + continue converted[key] = _json_ready(item) return converted if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py index 09e2c90e..ec40a575 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/acs_transfer.py @@ -44,7 +44,8 @@ has_support_role_metadata, support_role_series, ) -from microcosm.fit import QRFChainState +from microcosm.fit import DEFAULT_ZERO_ATOL, QRFChainState +from microcosm.fit.qrf import detect_regime from microcosm.frame import EntitySchema, Frame, Weights QRF: Any | None = None @@ -67,6 +68,7 @@ "TargetFamilies", "acs_transfer_donor_requirements", "acs_transfer_execution_contract_identity", + "acs_adult_care_qualifying_rows", "assert_acs_transfer_targets_are_input_leaves", "declared_acs_transfer_target_families", "default_acs_transfer_target_families", @@ -413,6 +415,10 @@ class AcsTransferPattern: weight_kind: str donor_rows: int recipient_rows: int + #: Ordered ``(model_target, regime)`` pairs explicitly requested by the + #: transfer owner and detected from this pattern's exact encoded donors. + #: Ordinary callers opt out, preserving the legacy provenance surface. + target_regimes: tuple[tuple[str, str], ...] = () @dataclass(frozen=True) @@ -651,6 +657,32 @@ def derive_acs_schedule_d_capital_gain_distributions( return values.astype(np.float64), provenance +def acs_adult_care_qualifying_rows(person: pd.DataFrame) -> pd.Series: + """Return the section 21 qualifying-person mask for ACS adult care. + + An incapable dependent qualifies directly. An incapable tax-unit head or + spouse qualifies only when the unit contains a spouse row. Missing input + columns fail closed so callers cannot silently broaden the carrier set. + """ + + required = {_ADULT_CARE_FLAG, _ADULT_CARE_ROLE, _ADULT_CARE_UNIT} + missing = sorted(required - set(person.columns)) + if missing: + raise ValueError( + "ACS adult-care qualification is missing required person " + f"columns: {missing}." + ) + + flag = person[_ADULT_CARE_FLAG].fillna(False).astype(bool) + role = person[_ADULT_CARE_ROLE].astype(str) + units = person[_ADULT_CARE_UNIT] + is_dependent = role.eq("DEPENDENT") + is_head = role.eq("HEAD") + is_spouse = role.eq("SPOUSE") + unit_married = is_spouse.groupby(units).transform("any") + return flag & (is_dependent | ((is_head | is_spouse) & unit_married)) + + def reconcile_acs_adult_care( person: pd.DataFrame, *, @@ -670,7 +702,6 @@ def reconcile_acs_adult_care( introducing an additional carrier. """ - flag = person[_ADULT_CARE_FLAG].fillna(False).astype(bool) raw_expenses = pd.to_numeric(person[_ADULT_CARE_EXPENSE], errors="coerce") expenses = raw_expenses.fillna(0.0) if mutable_rows is None: @@ -682,13 +713,8 @@ def reconcile_acs_adult_care( "mutable_rows must be a one-dimensional mask aligned to person." ) mutable = pd.Series(mutable_array, index=person.index) - role = person[_ADULT_CARE_ROLE].astype(str) units = person[_ADULT_CARE_UNIT] - is_dependent = role.eq("DEPENDENT") - is_head = role.eq("HEAD") - is_spouse = role.eq("SPOUSE") - unit_married = is_spouse.groupby(units).transform("any") - qualifying = flag & (is_dependent | ((is_head | is_spouse) & unit_married)) + qualifying = acs_adult_care_qualifying_rows(person) mutable_positive = mutable & (expenses > 0.0) cleared_ineligible_mask = mutable_positive & ~qualifying @@ -890,6 +916,7 @@ def transfer_acs_inputs( target_bank: AcsTransferTargetBank | None = None, derive_schedule_d: bool = True, execution_contract: Mapping[str, object] | None = None, + regime_evidence_targets: Iterable[tuple[str, str]] = (), ) -> AcsTransferResult: """Impute requested missing leaves from ``donor`` onto ``recipient``. @@ -916,6 +943,11 @@ def transfer_acs_inputs( after each ordered model target, so a retry can continue without changing the monolithic chained-QRF result. The ordinary in-memory fit remains the default for library callers that do not request durable banking. + + ``regime_evidence_targets`` is an explicit ``(entity, target)`` audit + selection. Only those targets incur donor-regime detection, fitted-result + verification, and pattern provenance. The default is empty so an owner + cannot accidentally broaden every transfer's runtime or receipt contract. """ _validate_frames(recipient, donor) @@ -940,6 +972,13 @@ def transfer_acs_inputs( requested, max_targets_per_fit=max_targets_per_fit, ) + requested_target_keys = frozenset( + (entity, target) for entity, _family, targets in requested for target in targets + ) + selected_regime_evidence = _normalize_regime_evidence_targets( + regime_evidence_targets, + requested=requested_target_keys, + ) if not requested: return AcsTransferResult( frame=canonicalize_frame_string_dtypes( @@ -965,6 +1004,25 @@ def transfer_acs_inputs( assert_acs_transfer_targets_are_input_leaves(all_targets) active = _missing_target_families(requested, recipient=recipient) + if selected_regime_evidence: + requested_families = { + (entity, family): targets for entity, family, targets in requested + } + active = [ + ( + entity, + family, + ( + requested_families[(entity, family)] + if all( + (entity, target) in selected_regime_evidence + for target in requested_families[(entity, family)] + ) + else active_targets + ), + ) + for entity, family, active_targets in active + ] if not active: return AcsTransferResult( frame=canonicalize_frame_string_dtypes( @@ -990,6 +1048,9 @@ def transfer_acs_inputs( bank_target_indexes = {key: index for index, key in enumerate(ordered_bank_targets)} for entity, family, targets in active: + family_regime_evidence_targets = tuple( + target for target in targets if (entity, target) in selected_regime_evidence + ) recipient_table = recipient.table(entity) target_missing = { target: ( @@ -1009,6 +1070,7 @@ def transfer_acs_inputs( target_missing=target_missing, seed=seed, n_estimators=n_estimators, + regime_evidence_targets=family_regime_evidence_targets, ) else: fitted = _fit_family_patterns_banked( @@ -1026,7 +1088,13 @@ def transfer_acs_inputs( for model_target in _model_target_names(targets) }, total_targets=len(ordered_bank_targets), + regime_evidence_targets=family_regime_evidence_targets, ) + patterns_without_regimes = ( + tuple(replace(pattern, target_regimes=()) for pattern in fitted.patterns) + if family_regime_evidence_targets + else fitted.patterns + ) for target in targets: predicted = _prediction_values( fitted.predictions[target], @@ -1054,7 +1122,11 @@ def transfer_acs_inputs( predictors=fitted.predictors, seed=fitted.family_seed, weight_kind=fitted.weight_kind, - patterns=fitted.patterns, + patterns=( + fitted.patterns + if target in family_regime_evidence_targets + else patterns_without_regimes + ), imputed_recipient_rows=int(imputed.sum()), unmodeled_recipient_rows=int((missing_rows & ~imputed).sum()), ) @@ -1244,6 +1316,127 @@ def _model_target_names(targets: Sequence[str]) -> tuple[str, ...]: return tuple(model_targets) +def _selected_model_target_names( + family_targets: Sequence[str], + selected_targets: Sequence[str], +) -> tuple[str, ...]: + """Resolve an exported-target subset against the full family codec.""" + + family = tuple(family_targets) + family_set = set(family) + selected = set(selected_targets) + unknown = sorted(selected - family_set) + if unknown: + raise ValueError( + f"Selected ACS regime-evidence targets are outside the family: {unknown}." + ) + immigration_pair = set(_IMMIGRATION_STATUS_TARGETS) + selected_model_targets = { + ( + _IMMIGRATION_STATUS_MODEL_TARGET + if target in immigration_pair and immigration_pair.issubset(family_set) + else target + ) + for target in selected + } + return tuple( + target + for target in _model_target_names(family) + if target in selected_model_targets + ) + + +def _regime_evidence_model_targets( + *, + model_targets: Sequence[str], + target_encodings: Mapping[str, _TargetEncoding], + regime_evidence_targets: Sequence[str], +) -> tuple[str, ...]: + """Project selected exported leaves onto the fitted model-target order.""" + + selected = { + target_encodings[target].model_target for target in regime_evidence_targets + } + return tuple(target for target in model_targets if target in selected) + + +def _model_target_regimes( + model_frame: Frame, + *, + entity: str, + model_targets: Sequence[str], + zero_atol: float, +) -> tuple[tuple[str, str], ...]: + """Detect ordered regimes on the exact encoded donor fit surface.""" + + table = model_frame.table(entity) + return tuple( + ( + target, + detect_regime( + table[target].to_numpy(dtype=np.float64), + zero_atol=zero_atol, + ), + ) + for target in model_targets + ) + + +def _verify_fitted_target_regimes( + fitted: object, + *, + expected: tuple[tuple[str, str], ...], + entity: str, + family: str, + pattern: str, +) -> None: + """Verify a fitted QRF's reported regimes when its API exposes them.""" + + if not expected: + return + regimes = getattr(fitted, "regimes", None) + if not callable(regimes): + # Lightweight test doubles need only implement the fit/predict surface. + return + reported = regimes() + if not isinstance(reported, Mapping): + raise TypeError( + f"ACS transfer {entity!r}/{family!r}/{pattern!r} QRF regimes " + "must be a mapping." + ) + expected_targets = tuple(target for target, _regime in expected) + actual = tuple((target, reported.get(target)) for target in expected_targets) + if any(target not in reported for target in expected_targets) or actual != expected: + raise RuntimeError( + f"ACS transfer {entity!r}/{family!r}/{pattern!r} QRF reported " + f"regimes {actual!r}, expected exact donor-support regimes " + f"{expected!r}." + ) + + +def _verify_chain_target_regime( + result: object, + *, + expected: str, + entity: str, + family: str, + pattern: str, + model_target: str, +) -> None: + """Verify one targetwise chain result when it exposes regime evidence.""" + + reported = getattr(result, "regime", None) + if reported is None: + # Lightweight bank-path test doubles may omit QRF diagnostics. + return + if reported != expected: + raise RuntimeError( + f"ACS transfer {entity!r}/{family!r}/{pattern!r} target " + f"{model_target!r} reported regime {reported!r}, expected exact " + f"donor-support regime {expected!r}." + ) + + def _fit_family_patterns( donor: Frame, recipient: Frame, @@ -1254,6 +1447,7 @@ def _fit_family_patterns( target_missing: Mapping[str, np.ndarray], seed: int, n_estimators: int, + regime_evidence_targets: tuple[str, ...], ) -> _FamilyFit: _validate_donor_targets(donor, entity=entity, targets=targets) donor_table = donor.table(entity) @@ -1269,6 +1463,11 @@ def _fit_family_patterns( complete=target_complete, ) model_targets = _model_target_names(targets) + evidence_model_targets = _regime_evidence_model_targets( + model_targets=model_targets, + target_encodings=target_encodings, + regime_evidence_targets=regime_evidence_targets, + ) surface = _transfer_feature_surface( donor, recipient, @@ -1338,7 +1537,18 @@ def _fit_family_patterns( mask=donor_mask, ) resolved_kind = model_frame.resolve_weights(entity).kind.value - fitted = _qrf()(n_estimators=n_estimators, seed=pattern_seed).fit( + model = _qrf()(n_estimators=n_estimators, seed=pattern_seed) + target_regimes = ( + _model_target_regimes( + model_frame, + entity=entity, + model_targets=evidence_model_targets, + zero_atol=float(getattr(model, "zero_atol", DEFAULT_ZERO_ATOL)), + ) + if evidence_model_targets + else () + ) + fitted = model.fit( model_frame, list(predictors), list(model_targets), @@ -1350,6 +1560,14 @@ def _fit_family_patterns( f"weight kind {fitted.weight_kind!r}, expected the donor " f"Frame's {resolved_kind!r}." ) + if target_regimes: + _verify_fitted_target_regimes( + fitted, + expected=target_regimes, + entity=entity, + family=family, + pattern=pattern_name, + ) recipient_pattern = _encoded_predictor_frame( surface.recipient.iloc[recipient_positions], @@ -1376,6 +1594,7 @@ def _fit_family_patterns( weight_kind=fitted.weight_kind, donor_rows=donor_rows, recipient_rows=len(recipient_positions), + target_regimes=target_regimes, ) pattern_records.append(pattern_record) fit_records.append( @@ -1421,6 +1640,7 @@ def _fit_family_patterns_banked( target_bank: AcsTransferTargetBank, target_indexes: Mapping[str, int], total_targets: int, + regime_evidence_targets: tuple[str, ...], ) -> _FamilyFit: """Fit one family targetwise, resuming exact raw chained draws.""" @@ -1445,6 +1665,11 @@ def _fit_family_patterns_banked( raise AssertionError( "ACS transfer model-target ordering changed during encode." ) + evidence_model_targets = _regime_evidence_model_targets( + model_targets=model_targets, + target_encodings=target_encodings, + regime_evidence_targets=regime_evidence_targets, + ) surface = _transfer_feature_surface( donor, @@ -1511,6 +1736,16 @@ def _fit_family_patterns_banked( ) resolved_kind = model_frame.resolve_weights(entity).kind.value model = _qrf()(n_estimators=n_estimators, seed=pattern_seed) + target_regimes = ( + _model_target_regimes( + model_frame, + entity=entity, + model_targets=evidence_model_targets, + zero_atol=float(getattr(model, "zero_atol", DEFAULT_ZERO_ATOL)), + ) + if evidence_model_targets + else () + ) if not hasattr(model, "start_chain") or not hasattr(model, "fit_draw_next"): raise TypeError( "Banked ACS transfer requires a QRF with start_chain and " @@ -1536,6 +1771,7 @@ def _fit_family_patterns_banked( weight_kind=state.weight_kind, donor_rows=donor_rows, recipient_rows=len(recipient_positions), + target_regimes=target_regimes, ) contexts.append( _BankPatternContext( @@ -1616,6 +1852,16 @@ def _fit_family_patterns_banked( f"resolved weight kind {result.weight_kind!r}, expected " f"{pattern.weight_kind!r}." ) + expected_regime = dict(pattern.target_regimes).get(model_target) + if expected_regime is not None: + _verify_chain_target_regime( + result, + expected=expected_regime, + entity=entity, + family=family, + pattern=pattern.name, + model_target=model_target, + ) raw_draw[context.recipient_positions] = result.raw_draw _validate_prediction_values( pd.DataFrame( @@ -2265,6 +2511,47 @@ def _validate_fit_options( ) +def _normalize_regime_evidence_targets( + targets: Iterable[tuple[str, str]], + *, + requested: frozenset[tuple[str, str]], +) -> frozenset[tuple[str, str]]: + """Validate an explicit owner selection against the requested surface.""" + + if isinstance(targets, (str, bytes)): + raise TypeError( + "regime_evidence_targets must contain (entity, target) pairs, not a string." + ) + try: + items = tuple(targets) + except TypeError as exc: + raise TypeError( + "regime_evidence_targets must be an iterable of (entity, target) pairs." + ) from exc + malformed = [ + item + for item in items + if ( + not isinstance(item, tuple) + or len(item) != 2 + or any(not isinstance(value, str) or not value for value in item) + ) + ] + if malformed: + raise TypeError( + "regime_evidence_targets contains malformed (entity, target) " + f"pair(s): {malformed!r}." + ) + selected = frozenset(items) + unknown = sorted(selected - requested) + if unknown: + raise ValueError( + "regime_evidence_targets names target(s) outside the requested " + f"transfer surface: {unknown}." + ) + return selected + + def _normalize_target_families( families: TargetFamilies, *, diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py b/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py new file mode 100644 index 00000000..d2e01e37 --- /dev/null +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/post_transfer_calibration.py @@ -0,0 +1,1855 @@ +"""Deterministic two-part calibration for mutable US transfer draws. + +This module is deliberately provenance-blind. Its callers own the source +spine and pass exact reference, recipient, and mutable row masks. The kernel +then calibrates only the positive leg: carrier prevalence is either matched to +the weighted reference margin or frozen, and mutable positive amounts are +mapped onto reference support at the terminal battery's five quantiles. +""" + +from __future__ import annotations + +import hashlib +import json +from collections.abc import Mapping +from dataclasses import asdict, dataclass +from types import MappingProxyType +from typing import Literal + +import numpy as np +import pandas as pd + +from microcosm.frame import Frame + +__all__ = [ + "POST_TRANSFER_CALIBRATION_SPECS", + "PostTransferCalibrationFrameResult", + "PostTransferCalibrationResult", + "PostTransferCalibrationSpec", + "apply_post_transfer_calibration", + "calibrate_post_transfer_values", + "post_transfer_calibration_policy_identity", + "post_transfer_calibration_spec", + "post_transfer_calibration_spec_for_target", + "validate_post_transfer_calibration_receipt", +] + +_QUANTILES = (0.10, 0.25, 0.50, 0.75, 0.90) +_POLICY_ARTIFACT_KIND = "microcosm_us_post_transfer_calibration_policy" +_POLICY_SCHEMA_VERSION = 1 +_RECEIPT_SCHEMA_VERSION = 2 + + +def _receipt_verification_contract() -> dict[str, object]: + """Describe which receipt claims can be replayed from a terminal frame.""" + + return { + "terminal_pre_state_replay": False, + "terminal_pre_state_reason": ( + "pre-calibration frame is not present at terminal validation" + ), + "terminal_live_receipt_paths": [ + "scope.rows", + "scope.reference_rows", + "scope.recipient_rows", + "scope.reference_rows_sha256", + "scope.recipient_rows_sha256", + "scope.entity_ids_sha256", + "scope.output_values_sha256", + "weights.sha256", + "weights.reference_total", + "weights.recipient_total", + "carrier.reference_positive_mass", + "carrier.reference_positive_share", + "carrier.target_positive_mass", + "carrier.after_positive_mass", + "carrier.after_positive_share", + "carrier.residual_after_minus_target", + "carrier.absolute_residual", + "amount.reference_quantiles", + "amount.recipient_after_quantiles", + "amount.qed_after", + ], + "generation_transition_receipt_paths": [ + "scope.mutable_rows", + "scope.effective_mutable_rows", + "scope.mutable_rows_sha256", + "scope.allowed_carrier_rows", + "scope.allowed_carrier_rows_sha256", + "scope.allowed_carrier_rows_mode", + "scope.addition_candidate_rows", + "scope.addition_candidate_rows_sha256", + "scope.addition_candidate_rows_mode", + "scope.input_values_sha256", + "carrier.before_positive_mass", + "carrier.before_positive_share", + "carrier.removed_rows", + "carrier.added_rows", + "carrier.disallowed_cleared_rows", + "carrier.capacity_limited", + "carrier.capacity", + "carrier.selection", + "amount.recipient_before_quantiles", + "amount.qed_before", + "amount.mapped_rows", + "amount.anchor_rows", + "amount.anchor_conflicts", + "amount.donor_support_violations", + "invariants", + ], + "frame_owner_only_receipt_paths": [ + "scope.applied_changed_rows", + "weights.kind", + ], + "policy_structural_receipt_paths": [ + "artifact_kind", + "schema_version", + "verification_contract", + "policy_sha256", + "spec", + "carrier.mode", + "amount.quantiles", + "amount.exact_anchor_count", + "amount.unanchored_quantiles", + "amount.status", + ], + "generation_transition_binding": ( + "generation-time input/output context and enclosing execution authority" + ), + } + + +@dataclass(frozen=True) +class PostTransferCalibrationSpec: + """One authority-bound positive-leg post-transfer calibration.""" + + entity: str + family: str + target: str + stage: Literal["early_gap_fill", "late_transfer"] + carrier_mode: Literal["match_reference", "preserve_recipient"] + negative_leg: Literal["byte_exact"] = "byte_exact" + special_constraint: Literal[ + "none", + "adult_care_qualifying_one_per_tax_unit", + "weeks_requires_positive_unemployment_compensation", + ] = "none" + + def __post_init__(self) -> None: + for label, value in ( + ("entity", self.entity), + ("family", self.family), + ("target", self.target), + ): + if not isinstance(value, str) or not value.strip(): + raise ValueError( + f"Post-transfer calibration {label} must be non-empty." + ) + if self.stage not in {"early_gap_fill", "late_transfer"}: + raise ValueError(f"Unknown post-transfer calibration stage {self.stage!r}.") + if self.carrier_mode not in {"match_reference", "preserve_recipient"}: + raise ValueError( + f"Unknown post-transfer carrier mode {self.carrier_mode!r}." + ) + if self.negative_leg != "byte_exact": + raise ValueError("Post-transfer calibration must preserve negatives.") + if self.special_constraint not in { + "none", + "adult_care_qualifying_one_per_tax_unit", + "weeks_requires_positive_unemployment_compensation", + }: + raise ValueError( + "Unknown post-transfer calibration special constraint " + f"{self.special_constraint!r}." + ) + + @property + def key(self) -> str: + return f"{self.entity}/{self.family}/{self.target}" + + +def _spec( + entity: str, + family: str, + target: str, + stage: Literal["early_gap_fill", "late_transfer"], + carrier_mode: Literal["match_reference", "preserve_recipient"], + special_constraint: Literal[ + "none", + "adult_care_qualifying_one_per_tax_unit", + "weeks_requires_positive_unemployment_compensation", + ] = "none", +) -> PostTransferCalibrationSpec: + return PostTransferCalibrationSpec( + entity=entity, + family=family, + target=target, + stage=stage, + carrier_mode=carrier_mode, + special_constraint=special_constraint, + ) + + +_ORDERED_SPECS = ( + _spec( + "person", + "model_required_numeric", + "unemployment_compensation", + "early_gap_fill", + "preserve_recipient", + ), + _spec( + "person", + "source_operator_prior_year_income", + "self_employment_income_last_year", + "early_gap_fill", + "match_reference", + ), + _spec( + "person", + "adult_care", + "pre_subsidy_care_expenses", + "late_transfer", + "match_reference", + "adult_care_qualifying_one_per_tax_unit", + ), + _spec( + "person", + "source_operator_child_support", + "child_support_expense", + "late_transfer", + "match_reference", + ), + _spec( + "person", + "source_operator_child_support", + "child_support_received", + "late_transfer", + "match_reference", + ), + _spec( + "person", + "source_operator_disability_benefits", + "disability_benefits", + "late_transfer", + "preserve_recipient", + ), + _spec( + "person", + "source_operator_weeks_unemployed", + "weeks_unemployed", + "late_transfer", + "match_reference", + "weeks_requires_positive_unemployment_compensation", + ), + _spec( + "person", + "source_operator_workers_compensation", + "workers_compensation", + "late_transfer", + "match_reference", + ), + _spec( + "spm_unit", + "source_operator_energy_subsidy", + "spm_unit_energy_subsidy", + "late_transfer", + "match_reference", + ), +) + +POST_TRANSFER_CALIBRATION_SPECS = MappingProxyType( + {spec.key: spec for spec in _ORDERED_SPECS} +) + + +@dataclass(frozen=True) +class PostTransferCalibrationResult: + """Calibrated values plus a deterministic audit receipt.""" + + values: np.ndarray + receipt: dict[str, object] + + +@dataclass(frozen=True) +class PostTransferCalibrationFrameResult: + """A frame with one calibrated target plus its audit receipt.""" + + frame: Frame + receipt: dict[str, object] + + +@dataclass(frozen=True) +class _PrefixSchedule: + """One bound carrier order and its sole float64 accumulation path.""" + + ordered_positions: np.ndarray + cumulative_mass: np.ndarray + + +def _canonical_sha256(value: object) -> str: + return hashlib.sha256( + json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + + +def post_transfer_calibration_policy_identity() -> dict[str, object]: + """Return the complete JSON policy identity, including its content hash.""" + + live_specs: list[PostTransferCalibrationSpec] = [] + if not isinstance(POST_TRANSFER_CALIBRATION_SPECS, Mapping): + raise ValueError("Post-transfer calibration registry must be a mapping.") + for key, spec in POST_TRANSFER_CALIBRATION_SPECS.items(): + if not isinstance(spec, PostTransferCalibrationSpec): + raise ValueError("Post-transfer calibration registry values must be specs.") + if key != spec.key: + raise ValueError( + "Post-transfer calibration registry key/spec mismatch: " + f"{key!r} != {spec.key!r}." + ) + live_specs.append(spec) + + payload: dict[str, object] = { + "artifact_kind": _POLICY_ARTIFACT_KIND, + "schema_version": _POLICY_SCHEMA_VERSION, + "scope": { + "reference": "asec_origin_clone_0", + "recipient": "acs_origin_clone_0", + "mutable": "caller_supplied_target_cells", + "provenance_masks": "caller_supplied_no_internal_inference", + "constraint_masks": "caller_supplied_hash_bound", + "value_dtype": "float64_byte_contract", + "zero_weight_rows": "byte_exact", + }, + "quantiles": list(_QUANTILES), + "carrier_selection": { + "match_reference": "weighted_positive_prevalence_nearest_prefix", + "removal_order": "positive_amount_descending_then_entity_id", + "addition_order": "entity_id", + "equal_distance": "lower_mass", + }, + "amount_mapping": { + "leg": "positive", + "recipient_rank": "weighted_full_recipient_positive_upper_cdf", + "inverse_cdf": "left", + "exact_quantile_anchors": list(_QUANTILES), + "infeasible_anchor_handling": "frame_owner_fail_closed", + "output_support": "reference_positive_values_only", + }, + "targets": [ + asdict(spec) for spec in sorted(live_specs, key=lambda item: item.key) + ], + } + return {**payload, "sha256": _canonical_sha256(payload)} + + +def post_transfer_calibration_spec( + *, + entity: str, + family: str, + target: str, +) -> PostTransferCalibrationSpec: + """Resolve one exact declared calibration spec.""" + + key = f"{entity}/{family}/{target}" + try: + return POST_TRANSFER_CALIBRATION_SPECS[key] + except KeyError as exc: + raise ValueError( + f"No post-transfer calibration is declared for {key}." + ) from exc + + +def post_transfer_calibration_spec_for_target( + *, + entity: str, + target: str, +) -> PostTransferCalibrationSpec: + """Resolve an entity/target only when its declared family is unambiguous.""" + + matches = [ + spec + for spec in POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.entity == entity and spec.target == target + ] + if len(matches) != 1: + raise ValueError( + "Post-transfer calibration entity/target lookup must resolve exactly " + f"one spec; {entity}/{target} resolved {len(matches)}." + ) + return matches[0] + + +def _require_declared_calibration_spec( + spec: PostTransferCalibrationSpec, +) -> None: + """Reject caller-constructed specs outside the live authority registry.""" + + declared = POST_TRANSFER_CALIBRATION_SPECS.get(spec.key) + if declared != spec: + raise ValueError( + "Post-transfer calibration spec is not the exact live declared " + f"policy entry for {spec.key}." + ) + + +def _aligned_bool_mask( + values: object, + *, + size: int, + label: str, +) -> np.ndarray: + mask = np.asarray(values) + if mask.ndim != 1 or mask.shape != (size,) or mask.dtype.kind != "b": + raise ValueError( + f"{label} must be a one-dimensional boolean mask of length {size}." + ) + return mask.astype(bool, copy=True) + + +def _numeric_values(values: object, *, label: str) -> np.ndarray: + series = pd.Series(values, copy=False) + numeric = pd.to_numeric(series, errors="coerce").to_numpy(dtype=np.float64) + if numeric.ndim != 1: + raise ValueError(f"{label} must be one-dimensional.") + return numeric + + +def _float64_values(values: object, *, label: str) -> np.ndarray: + """Return an exact float64 vector or reject a lossy representation change.""" + + array = np.asarray(values) + if array.ndim != 1: + raise ValueError(f"{label} must be one-dimensional.") + if array.dtype != np.dtype(np.float64): + raise ValueError( + f"{label} must have exact float64 dtype so byte invariants are " + f"meaningful; got {array.dtype}." + ) + return array.copy() + + +def _json_scalar(value: object) -> object: + if isinstance(value, np.generic): + return value.item() + if value is None or isinstance(value, (str, int, float, bool)): + return value + return repr(value) + + +def _stable_id_order(entity_ids: np.ndarray, positions: np.ndarray) -> np.ndarray: + selected = entity_ids[positions] + try: + order = np.argsort(selected, kind="stable") + except TypeError: + tokens = np.asarray( + [ + f"{type(value).__module__}.{type(value).__qualname__}:" + f"{_json_scalar(value)!r}" + for value in selected + ], + dtype=object, + ) + order = np.argsort(tokens, kind="stable") + return positions[order] + + +def _descending_value_then_id_order( + values: np.ndarray, + entity_ids: np.ndarray, + positions: np.ndarray, +) -> np.ndarray: + by_id = _stable_id_order(entity_ids, positions) + return by_id[np.argsort(-values[by_id], kind="stable")] + + +def _prefix_schedule( + ordered_positions: np.ndarray, + weights: np.ndarray, +) -> _PrefixSchedule: + positions = np.asarray(ordered_positions, dtype=np.int64).copy() + cumulative = np.empty(len(positions) + 1, dtype=np.float64) + cumulative[0] = 0.0 + np.cumsum(weights[positions], dtype=np.float64, out=cumulative[1:]) + positions.setflags(write=False) + cumulative.setflags(write=False) + return _PrefixSchedule( + ordered_positions=positions, + cumulative_mass=cumulative, + ) + + +def _nearest_prefix( + schedule: _PrefixSchedule, + target_mass: float, +) -> tuple[np.ndarray, float, dict[str, object]]: + ordered_positions = schedule.ordered_positions + cumulative = schedule.cumulative_mass + requested = max(0.0, target_mass) + # cumulative is ascending, so np.argmin implements the declared lower-mass + # tie break when two adjacent prefixes are equally close. + take = int(np.argmin(np.abs(cumulative - requested))) + lower_index = max( + 0, + min( + int(np.searchsorted(cumulative, requested, side="right")) - 1, + len(cumulative) - 1, + ), + ) + upper_index = min(lower_index + 1, len(cumulative) - 1) + chosen_mass = float(cumulative[take]) + audit: dict[str, object] = { + "requested_prefix_mass": float(requested), + "candidate_rows": int(len(ordered_positions)), + "candidate_mass": float(cumulative[-1]), + "chosen_prefix_rows": take, + "chosen_prefix_mass": chosen_mass, + "lower_prefix_mass": float(cumulative[lower_index]), + "upper_prefix_mass": float(cumulative[upper_index]), + } + return ordered_positions[:take], chosen_mass, audit + + +def _weighted_inverse_quantiles( + values: np.ndarray, + weights: np.ndarray, + probabilities: tuple[float, ...] = _QUANTILES, +) -> np.ndarray | None: + included = (values > 0.0) & (weights > 0.0) + if not included.any(): + return None + positions = np.flatnonzero(included) + order = positions[np.argsort(values[positions], kind="stable")] + cumulative = np.cumsum(weights[order], dtype=np.float64) + cumulative /= cumulative[-1] + indices = np.minimum( + np.searchsorted(cumulative, np.asarray(probabilities), side="left"), + len(order) - 1, + ) + return values[order[indices]] + + +def _qed(left: np.ndarray | None, right: np.ndarray | None) -> float | None: + if left is None or right is None: + return None + denominator = np.abs(left) + np.abs(right) + distances = np.divide( + 2.0 * np.abs(left - right), + denominator, + out=np.zeros_like(denominator), + where=denominator > 0.0, + ) + return float(np.max(distances)) + + +def _float_list(values: np.ndarray | None) -> list[float] | None: + return None if values is None else [float(value) for value in values] + + +def _is_finite_number(value: object) -> bool: + """Return whether a JSON scalar is a finite real without raising.""" + + if not isinstance(value, (int, float)) or isinstance(value, bool): + return False + try: + return bool(np.isfinite(float(value))) + except (OverflowError, TypeError, ValueError): + return False + + +def _selected_bytes(values: np.ndarray, mask: np.ndarray) -> bytes: + return np.ascontiguousarray(values[mask]).tobytes(order="C") + + +def _mask_sha256(mask: np.ndarray) -> str: + return hashlib.sha256(np.ascontiguousarray(mask).tobytes(order="C")).hexdigest() + + +def _values_sha256(values: np.ndarray) -> str: + return hashlib.sha256(np.ascontiguousarray(values).tobytes(order="C")).hexdigest() + + +def _map_positive_amounts( + working: np.ndarray, + *, + original: np.ndarray, + weights: np.ndarray, + entity_ids: np.ndarray, + reference_rows: np.ndarray, + recipient_rows: np.ndarray, + mutable_effective: np.ndarray, + final_mutable_carriers: np.ndarray, +) -> dict[str, object]: + donor_positions = np.flatnonzero( + reference_rows & (weights > 0.0) & (original > 0.0) + ) + donor_positions = _stable_id_order(entity_ids, donor_positions) + donor_positions = donor_positions[ + np.argsort(original[donor_positions], kind="stable") + ] + donor_values = original[donor_positions] + donor_weights = weights[donor_positions] + donor_cumulative = np.cumsum(donor_weights, dtype=np.float64) + donor_cumulative /= donor_cumulative[-1] + + recipient_carrier_rows = ( + recipient_rows & (weights > 0.0) & ((working > 0.0) | final_mutable_carriers) + ) + recipient_positions = np.flatnonzero(recipient_carrier_rows) + mutable_positions = np.flatnonzero(recipient_carrier_rows & mutable_effective) + before_quantiles = _weighted_inverse_quantiles( + original[recipient_rows], weights[recipient_rows] + ) + reference_quantiles = _weighted_inverse_quantiles( + original[reference_rows], weights[reference_rows] + ) + anchor_conflicts: list[dict[str, object]] = [] + anchor_rows: list[dict[str, object]] = [] + if recipient_positions.size: + recipient_positions = _stable_id_order(entity_ids, recipient_positions) + recipient_positions = recipient_positions[ + np.argsort(working[recipient_positions], kind="stable") + ] + recipient_cumulative = np.cumsum(weights[recipient_positions], dtype=np.float64) + recipient_cumulative /= recipient_cumulative[-1] + donor_indices = np.minimum( + np.searchsorted(donor_cumulative, recipient_cumulative, side="left"), + len(donor_positions) - 1, + ) + mapped = donor_values[donor_indices] + mutable_in_order = mutable_effective[recipient_positions] + working[recipient_positions[mutable_in_order]] = mapped[mutable_in_order] + + # Resolve anchors against the complete recipient positive CDF, including + # immutable carriers. A sparse or immutable recipient can make two + # distinct donor anchors compete for one row; that condition is recorded + # explicitly and is rejected by the production owner. + recipient_positions = _stable_id_order(entity_ids, recipient_positions) + recipient_positions = recipient_positions[ + np.argsort(working[recipient_positions], kind="stable") + ] + recipient_cumulative = np.cumsum(weights[recipient_positions], dtype=np.float64) + recipient_cumulative /= recipient_cumulative[-1] + occupied: dict[int, tuple[float, float]] = {} + for probability in _QUANTILES: + recipient_rank = min( + int(np.searchsorted(recipient_cumulative, probability, side="left")), + len(recipient_positions) - 1, + ) + donor_rank = min( + int(np.searchsorted(donor_cumulative, probability, side="left")), + len(donor_positions) - 1, + ) + donor_value = float(donor_values[donor_rank]) + previous = occupied.get(recipient_rank) + recipient_position = recipient_positions[recipient_rank] + mutable_anchor = bool(mutable_effective[recipient_position]) + conflict_reason: str | None = None + if previous is not None and previous[1] != donor_value: + conflict_reason = "distinct_reference_values_share_recipient_row" + elif not mutable_anchor and working[recipient_position] != donor_value: + conflict_reason = "recipient_anchor_row_is_immutable" + if conflict_reason is not None: + anchor_conflicts.append( + { + "reason": conflict_reason, + "recipient_rank": recipient_rank, + "earlier_quantile": (None if previous is None else previous[0]), + "earlier_reference_value": ( + None if previous is None else previous[1] + ), + "quantile": probability, + "reference_value": donor_value, + } + ) + occupied[recipient_rank] = (probability, donor_value) + if mutable_anchor: + working[recipient_position] = donor_value + anchor_rows.append( + { + "quantile": probability, + "entity_id": _json_scalar(entity_ids[recipient_position]), + "recipient_rank": recipient_rank, + "reference_value": donor_value, + "mutable": mutable_anchor, + } + ) + + after_quantiles = _weighted_inverse_quantiles( + working[recipient_rows], weights[recipient_rows] + ) + unanchored: list[float] = ( + list(_QUANTILES) + if reference_quantiles is not None and after_quantiles is None + else [] + ) + if reference_quantiles is not None and after_quantiles is not None: + unanchored = [ + probability + for probability, reference_value, recipient_value in zip( + _QUANTILES, + reference_quantiles, + after_quantiles, + strict=True, + ) + if recipient_value != reference_value + ] + mapped_values = working[mutable_positions] + support_violations = int( + np.count_nonzero(~np.isin(mapped_values, donor_values, assume_unique=False)) + ) + return { + "quantiles": list(_QUANTILES), + "reference_quantiles": _float_list(reference_quantiles), + "recipient_before_quantiles": _float_list(before_quantiles), + "recipient_after_quantiles": _float_list(after_quantiles), + "qed_before": _qed(reference_quantiles, before_quantiles), + "qed_after": _qed(reference_quantiles, after_quantiles), + "mapped_rows": int(mutable_positions.size), + "anchor_rows": anchor_rows, + "exact_anchor_count": len(_QUANTILES) - len(unanchored), + "anchor_conflicts": anchor_conflicts, + "unanchored_quantiles": unanchored, + "donor_support_violations": support_violations, + "status": ( + "exact" + if not anchor_conflicts and not unanchored + else "infeasible_exact_anchors" + ), + } + + +def calibrate_post_transfer_values( + values: object, + weights: object, + entity_ids: object, + *, + spec: PostTransferCalibrationSpec, + reference_rows: object, + recipient_rows: object, + mutable_rows: object, + allowed_carrier_rows: object | None = None, + addition_candidate_rows: object | None = None, +) -> PostTransferCalibrationResult: + """Calibrate one positive transfer leg without reading provenance columns.""" + + if not isinstance(spec, PostTransferCalibrationSpec): + raise TypeError("spec must be a PostTransferCalibrationSpec.") + _require_declared_calibration_spec(spec) + if spec.special_constraint != "none" and ( + allowed_carrier_rows is None or addition_candidate_rows is None + ): + raise ValueError( + f"Post-transfer calibration {spec.key} requires explicit " + "allowed_carrier_rows and addition_candidate_rows for special " + f"constraint {spec.special_constraint!r}." + ) + original = _float64_values(values, label="values") + size = len(original) + numeric_weights = _numeric_values(weights, label="weights") + ids = np.asarray(entity_ids) + if len(numeric_weights) != size or ids.ndim != 1 or len(ids) != size: + raise ValueError("values, weights, and entity_ids must align one-to-one.") + if not np.isfinite(numeric_weights).all() or (numeric_weights < 0.0).any(): + raise ValueError( + "Post-transfer calibration weights must be finite and nonnegative." + ) + if pd.Series(ids).isna().any() or pd.Series(ids).duplicated().any(): + raise ValueError( + "Post-transfer calibration entity_ids must be complete and unique." + ) + + reference = _aligned_bool_mask(reference_rows, size=size, label="reference_rows") + recipient = _aligned_bool_mask(recipient_rows, size=size, label="recipient_rows") + mutable = _aligned_bool_mask(mutable_rows, size=size, label="mutable_rows") + if (reference & recipient).any(): + raise ValueError("reference_rows and recipient_rows must be disjoint.") + if (mutable & ~recipient).any(): + raise ValueError("mutable_rows must be a subset of recipient_rows.") + if not (reference & (numeric_weights > 0.0)).any(): + raise ValueError("Reference rows have no positive-weight support.") + if not (recipient & (numeric_weights > 0.0)).any(): + raise ValueError("Recipient rows have no positive-weight support.") + relevant = (reference | recipient) & (numeric_weights > 0.0) + if not np.isfinite(original[relevant]).all(): + raise ValueError( + "Positive-weight reference and recipient values must be finite." + ) + if not (reference & (numeric_weights > 0.0) & (original > 0.0)).any(): + raise ValueError("Reference rows have no positive donor support.") + + mutable_effective = mutable & (numeric_weights > 0.0) + allowed = ( + mutable.copy() + if allowed_carrier_rows is None + else _aligned_bool_mask( + allowed_carrier_rows, + size=size, + label="allowed_carrier_rows", + ) + ) + additions = ( + allowed.copy() + if addition_candidate_rows is None + else _aligned_bool_mask( + addition_candidate_rows, + size=size, + label="addition_candidate_rows", + ) + ) + for label, mask in ( + ("allowed_carrier_rows", allowed), + ("addition_candidate_rows", additions), + ): + if (mask & ~(mutable & recipient)).any(): + raise ValueError(f"{label} must be a subset of mutable recipient rows.") + if (additions & ~allowed).any(): + raise ValueError("addition_candidate_rows must be a subset of allowed rows.") + + protected = ~mutable_effective + negative = original < 0.0 + negative_zero = (original == 0.0) & np.signbit(original) + zero_weight = numeric_weights == 0.0 + protected_bytes = _selected_bytes(original, protected) + negative_bytes = _selected_bytes(original, negative) + negative_zero_bytes = _selected_bytes(original, negative_zero) + zero_weight_bytes = _selected_bytes(original, zero_weight) + + working = original.copy() + reference_total = float(numeric_weights[reference].sum()) + recipient_weights = numeric_weights[recipient] + recipient_total = float(recipient_weights.sum()) + reference_positive_mass = float(numeric_weights[reference & (original > 0.0)].sum()) + reference_share = reference_positive_mass / reference_total + target_positive_mass = reference_share * recipient_total + before_positive = recipient & (original > 0.0) + before_positive_mass = float(numeric_weights[before_positive].sum()) + before_carriers = before_positive.copy() + removed = np.zeros(size, dtype=bool) + added = np.zeros(size, dtype=bool) + disallowed = np.zeros(size, dtype=bool) + capacity_limited = False + capacity_receipt: dict[str, object] | None = None + selection_receipt: dict[str, object] | None = None + + if spec.carrier_mode == "match_reference": + disallowed = mutable_effective & (working > 0.0) & ~allowed + working[disallowed] = 0.0 + removed |= disallowed + fixed_positive = recipient & (working > 0.0) & ~mutable_effective + fixed_mass = float(numeric_weights[fixed_positive].sum()) + allowed_positive = mutable_effective & allowed & (working > 0.0) + allowed_positive_order = _descending_value_then_id_order( + working, + ids, + np.flatnonzero(allowed_positive), + ) + allowed_positive_schedule = _prefix_schedule( + allowed_positive_order, + numeric_weights, + ) + allowed_positive_mass = float(allowed_positive_schedule.cumulative_mass[-1]) + zero_candidates = ( + mutable_effective + & allowed + & additions + & (working == 0.0) + & ~np.signbit(working) + ) + addition_candidate_order = _stable_id_order( + ids, + np.flatnonzero(zero_candidates), + ) + addition_candidate_schedule = _prefix_schedule( + addition_candidate_order, + numeric_weights, + ) + addition_candidate_mass = float(addition_candidate_schedule.cumulative_mass[-1]) + minimum_attainable_mass = fixed_mass + # The ceiling is the mass of one declared set, not the scalar sum of + # independently accumulated partition endpoints. Retaining the + # recipient vector's length and order gives this nonnegative subset + # the same reduction topology as recipient_total, so the exact + # subset relationship is structural under floating-point addition. + attainable_rows = fixed_positive | allowed_positive | zero_candidates + attainable_recipient_weights = np.where( + attainable_rows[recipient], + recipient_weights, + 0.0, + ) + maximum_attainable_mass = float(attainable_recipient_weights.sum()) + capacity_limited = bool( + target_positive_mass < minimum_attainable_mass + or target_positive_mass > maximum_attainable_mass + ) + desired_mutable_mass = max(0.0, target_positive_mass - fixed_mass) + if desired_mutable_mass <= allowed_positive_mass: + retained, retained_mass, prefix_audit = _nearest_prefix( + allowed_positive_schedule, + desired_mutable_mass, + ) + retained_mask = np.zeros(size, dtype=bool) + retained_mask[retained] = True + clear = allowed_positive & ~retained_mask + working[clear] = 0.0 + removed |= clear + selection_receipt = { + "action": "retain_positive_prefix", + "base_positive_mass": fixed_mass, + "selected_rows": int(len(retained)), + "selected_mass": retained_mass, + **prefix_audit, + } + else: + needed = desired_mutable_mass - allowed_positive_mass + selected, selected_mass, prefix_audit = _nearest_prefix( + addition_candidate_schedule, + needed, + ) + added[selected] = True + selection_receipt = { + "action": "add_zero_prefix", + "base_positive_mass": fixed_mass + allowed_positive_mass, + "selected_rows": int(len(selected)), + "selected_mass": selected_mass, + **prefix_audit, + } + capacity_receipt = { + "fixed_positive_rows": int(fixed_positive.sum()), + "fixed_positive_mass": fixed_mass, + "allowed_positive_rows_before": int(allowed_positive.sum()), + "allowed_positive_mass_before": allowed_positive_mass, + "addition_candidate_rows": int(zero_candidates.sum()), + "addition_candidate_mass": addition_candidate_mass, + "minimum_attainable_mass": minimum_attainable_mass, + "maximum_attainable_mass": maximum_attainable_mass, + "target_within_attainable_interval": not capacity_limited, + } + + final_mutable_carriers = mutable_effective & ((working > 0.0) | added) + amount_receipt = _map_positive_amounts( + working, + original=original, + weights=numeric_weights, + entity_ids=ids, + reference_rows=reference, + recipient_rows=recipient, + mutable_effective=mutable_effective, + final_mutable_carriers=final_mutable_carriers, + ) + after_positive = recipient & (working > 0.0) + after_positive_mass = float(numeric_weights[after_positive].sum()) + after_share = after_positive_mass / recipient_total + + if spec.carrier_mode == "match_reference": + assert capacity_receipt is not None + assert selection_receipt is not None + minimum = float(capacity_receipt["minimum_attainable_mass"]) + maximum = float(capacity_receipt["maximum_attainable_mass"]) + if target_positive_mass < minimum: + saturated = np.isclose(after_positive_mass, minimum, rtol=1e-12, atol=1e-12) + elif target_positive_mass > maximum: + saturated = np.isclose(after_positive_mass, maximum, rtol=1e-12, atol=1e-12) + else: + saturated = True + capacity_receipt["capacity_boundary_saturated"] = bool(saturated) + if not saturated: + raise ValueError( + "Post-transfer carrier calibration did not saturate its " + "attainable-mass boundary." + ) + + immutable_ok = _selected_bytes(working, protected) == protected_bytes + negative_ok = _selected_bytes(working, negative) == negative_bytes + negative_zero_ok = _selected_bytes(working, negative_zero) == negative_zero_bytes + zero_weight_ok = _selected_bytes(working, zero_weight) == zero_weight_bytes + preserve_carriers = bool( + np.array_equal(before_carriers[recipient], after_positive[recipient]) + ) + allowed_violations = int( + (after_positive & mutable_effective & ~allowed).sum() + if spec.carrier_mode == "match_reference" + else 0 + ) + exact_quantile_anchors = bool( + amount_receipt["status"] == "exact" + and amount_receipt["exact_anchor_count"] == len(_QUANTILES) + and not amount_receipt["anchor_conflicts"] + and not amount_receipt["unanchored_quantiles"] + ) + if not all( + ( + immutable_ok, + negative_ok, + negative_zero_ok, + zero_weight_ok, + amount_receipt["donor_support_violations"] == 0, + allowed_violations == 0, + ) + ): + raise ValueError("Post-transfer calibration violated a byte/support invariant.") + if spec.carrier_mode == "preserve_recipient" and not preserve_carriers: + raise ValueError("Preserve-recipient calibration changed carrier membership.") + + policy = post_transfer_calibration_policy_identity() + receipt: dict[str, object] = { + "artifact_kind": "microcosm_us_post_transfer_calibration_receipt", + "schema_version": _RECEIPT_SCHEMA_VERSION, + "verification_contract": _receipt_verification_contract(), + "policy_sha256": policy["sha256"], + "spec": asdict(spec), + "scope": { + "rows": size, + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "mutable_rows": int(mutable.sum()), + "effective_mutable_rows": int(mutable_effective.sum()), + "reference_rows_sha256": _mask_sha256(reference), + "recipient_rows_sha256": _mask_sha256(recipient), + "mutable_rows_sha256": _mask_sha256(mutable), + "allowed_carrier_rows": int(allowed.sum()), + "allowed_carrier_rows_sha256": _mask_sha256(allowed), + "allowed_carrier_rows_mode": ( + "default_mutable" if allowed_carrier_rows is None else "caller_supplied" + ), + "addition_candidate_rows": int(additions.sum()), + "addition_candidate_rows_sha256": _mask_sha256(additions), + "addition_candidate_rows_mode": ( + "default_allowed" + if addition_candidate_rows is None + else "caller_supplied" + ), + "entity_ids_sha256": _canonical_sha256( + [_json_scalar(value) for value in ids.tolist()] + ), + "input_values_sha256": _values_sha256(original), + "output_values_sha256": _values_sha256(working), + }, + "weights": { + "sha256": _values_sha256(numeric_weights), + "reference_total": reference_total, + "recipient_total": recipient_total, + }, + "carrier": { + "mode": spec.carrier_mode, + "reference_positive_mass": reference_positive_mass, + "reference_positive_share": reference_share, + "target_positive_mass": target_positive_mass, + "before_positive_mass": before_positive_mass, + "before_positive_share": before_positive_mass / recipient_total, + "after_positive_mass": after_positive_mass, + "after_positive_share": after_share, + "residual_after_minus_target": after_positive_mass - target_positive_mass, + "absolute_residual": abs(after_positive_mass - target_positive_mass), + "removed_rows": int(removed.sum()), + "added_rows": int(added.sum()), + "disallowed_cleared_rows": int(disallowed.sum()), + "capacity_limited": capacity_limited, + "capacity": capacity_receipt, + "selection": selection_receipt, + }, + "amount": amount_receipt, + "invariants": { + "immutable_bytes_preserved": immutable_ok, + "negative_bytes_preserved": negative_ok, + "negative_zero_bytes_preserved": negative_zero_ok, + "zero_weight_bytes_preserved": zero_weight_ok, + "preserve_carriers": preserve_carriers, + "allowed_carrier_violations": allowed_violations, + "exact_quantile_anchors": exact_quantile_anchors, + }, + } + return PostTransferCalibrationResult( + values=working, + receipt={**receipt, "sha256": _canonical_sha256(receipt)}, + ) + + +def validate_post_transfer_calibration_receipt( + receipt: Mapping[str, object], + *, + spec: PostTransferCalibrationSpec, + boundary: str, + expected_policy_sha256: str | None = None, + expected_scope: Mapping[str, object] | None = None, + expected_weights_sha256: str | None = None, + require_exact_anchors: bool = True, +) -> None: + """Validate a receipt's schema and generation-time audit relationships. + + Terminal callers must separately replay the fields named by + ``verification_contract.terminal_live_receipt_paths`` against a live final + frame. Pre-calibration transition claims cannot be reconstructed from that + frame and are authenticated only by their generation context and enclosing + execution authority. + """ + + if not isinstance(spec, PostTransferCalibrationSpec): + raise TypeError("spec must be a PostTransferCalibrationSpec.") + _require_declared_calibration_spec(spec) + if not isinstance(receipt, Mapping): + raise ValueError(f"{boundary}: post-transfer calibration receipt is absent.") + payload = dict(receipt) + observed_sha256 = payload.pop("sha256", None) + if observed_sha256 != _canonical_sha256(payload): + raise ValueError( + f"{boundary}: post-transfer calibration receipt digest is invalid." + ) + policy_sha256 = ( + post_transfer_calibration_policy_identity()["sha256"] + if expected_policy_sha256 is None + else expected_policy_sha256 + ) + if ( + receipt.get("artifact_kind") != "microcosm_us_post_transfer_calibration_receipt" + or receipt.get("schema_version") != _RECEIPT_SCHEMA_VERSION + or receipt.get("verification_contract") != _receipt_verification_contract() + or receipt.get("policy_sha256") != policy_sha256 + or receipt.get("spec") != asdict(spec) + ): + raise ValueError( + f"{boundary}: post-transfer calibration policy/spec binding is invalid." + ) + scope = receipt.get("scope") + required_scope_digests = { + "reference_rows_sha256", + "recipient_rows_sha256", + "mutable_rows_sha256", + "allowed_carrier_rows_sha256", + "addition_candidate_rows_sha256", + "entity_ids_sha256", + "input_values_sha256", + "output_values_sha256", + } + required_scope_counts = { + "rows", + "reference_rows", + "recipient_rows", + "mutable_rows", + "effective_mutable_rows", + "allowed_carrier_rows", + "addition_candidate_rows", + } + required_scope_modes = { + "allowed_carrier_rows_mode", + "addition_candidate_rows_mode", + } + if not isinstance(scope, Mapping) or not ( + required_scope_digests | required_scope_counts | required_scope_modes + ).issubset(scope): + raise ValueError( + f"{boundary}: post-transfer calibration scope evidence is incomplete." + ) + if any( + not isinstance(scope.get(key), int) + or isinstance(scope.get(key), bool) + or scope[key] < 0 + for key in required_scope_counts + ): + raise ValueError( + f"{boundary}: post-transfer calibration scope counts are invalid." + ) + if ( + scope["reference_rows"] > scope["rows"] + or scope["recipient_rows"] > scope["rows"] + or scope["mutable_rows"] > scope["recipient_rows"] + or scope["effective_mutable_rows"] > scope["mutable_rows"] + or scope["allowed_carrier_rows"] > scope["mutable_rows"] + or scope["addition_candidate_rows"] > scope["allowed_carrier_rows"] + or scope["allowed_carrier_rows_mode"] + not in {"default_mutable", "caller_supplied"} + or scope["addition_candidate_rows_mode"] + not in {"default_allowed", "caller_supplied"} + ): + raise ValueError( + f"{boundary}: post-transfer calibration scope relationships are invalid." + ) + if spec.special_constraint != "none" and ( + scope["allowed_carrier_rows_mode"] != "caller_supplied" + or scope["addition_candidate_rows_mode"] != "caller_supplied" + ): + raise ValueError( + f"{boundary}: post-transfer special-constraint masks were not " + "explicitly supplied." + ) + digest_values = [scope.get(key) for key in required_scope_digests] + weights_receipt = receipt.get("weights") + if not isinstance(weights_receipt, Mapping): + raise ValueError( + f"{boundary}: post-transfer calibration weight evidence is absent." + ) + digest_values.append(weights_receipt.get("sha256")) + if any( + not isinstance(value, str) + or len(value) != 64 + or any(character not in "0123456789abcdef" for character in value) + for value in digest_values + ): + raise ValueError( + f"{boundary}: post-transfer calibration context digest is invalid." + ) + for total_key in ("reference_total", "recipient_total"): + value = weights_receipt.get(total_key) + if not _is_finite_number(value) or value <= 0.0: + raise ValueError( + f"{boundary}: post-transfer calibration weight totals are invalid." + ) + if expected_scope is not None and any( + scope.get(key) != value for key, value in expected_scope.items() + ): + raise ValueError( + f"{boundary}: post-transfer calibration scope does not match the " + "expected live context." + ) + if expected_weights_sha256 is not None and ( + weights_receipt.get("sha256") != expected_weights_sha256 + ): + raise ValueError( + f"{boundary}: post-transfer calibration weights do not match the " + "expected live context." + ) + invariants = receipt.get("invariants") + required_true = { + "immutable_bytes_preserved", + "negative_bytes_preserved", + "negative_zero_bytes_preserved", + "zero_weight_bytes_preserved", + } + if not isinstance(invariants, Mapping) or any( + invariants.get(key) is not True for key in required_true + ): + raise ValueError( + f"{boundary}: post-transfer calibration byte invariants are invalid." + ) + if invariants.get("allowed_carrier_violations") != 0: + raise ValueError( + f"{boundary}: post-transfer calibration carrier constraint failed." + ) + if spec.carrier_mode == "preserve_recipient" and ( + invariants.get("preserve_carriers") is not True + ): + raise ValueError( + f"{boundary}: post-transfer calibration changed frozen carriers." + ) + carrier = receipt.get("carrier") + required_carrier_masses = { + "reference_positive_mass", + "reference_positive_share", + "target_positive_mass", + "before_positive_mass", + "before_positive_share", + "after_positive_mass", + "after_positive_share", + "residual_after_minus_target", + "absolute_residual", + } + required_carrier_counts = { + "removed_rows", + "added_rows", + "disallowed_cleared_rows", + } + expected_carrier_keys = ( + required_carrier_masses + | required_carrier_counts + | {"mode", "capacity_limited", "capacity", "selection"} + ) + if ( + not isinstance(carrier, Mapping) + or carrier.get("mode") != spec.carrier_mode + or set(carrier) != expected_carrier_keys + or not isinstance(carrier.get("capacity_limited"), bool) + ): + raise ValueError( + f"{boundary}: post-transfer calibration carrier evidence is incomplete." + ) + if any( + not _is_finite_number(carrier.get(key)) for key in required_carrier_masses + ) or any( + not isinstance(carrier.get(key), int) + or isinstance(carrier.get(key), bool) + or carrier[key] < 0 + for key in required_carrier_counts + ): + raise ValueError( + f"{boundary}: post-transfer calibration carrier values are invalid." + ) + reference_total = float(weights_receipt["reference_total"]) + recipient_total = float(weights_receipt["recipient_total"]) + expected_carrier_values = { + "reference_positive_share": ( + float(carrier["reference_positive_mass"]) / reference_total + ), + "target_positive_mass": ( + float(carrier["reference_positive_share"]) * recipient_total + ), + "before_positive_share": ( + float(carrier["before_positive_mass"]) / recipient_total + ), + "after_positive_share": ( + float(carrier["after_positive_mass"]) / recipient_total + ), + "residual_after_minus_target": ( + float(carrier["after_positive_mass"]) + - float(carrier["target_positive_mass"]) + ), + "absolute_residual": abs(float(carrier["residual_after_minus_target"])), + } + if any( + not np.isclose( + float(carrier[key]), + expected, + rtol=1e-12, + atol=1e-12, + ) + for key, expected in expected_carrier_values.items() + ) or ( + float(carrier["reference_positive_mass"]) <= 0.0 + or float(carrier["reference_positive_mass"]) > reference_total + or not 0.0 < float(carrier["reference_positive_share"]) <= 1.0 + or float(carrier["target_positive_mass"]) <= 0.0 + or float(carrier["before_positive_mass"]) > recipient_total + or float(carrier["after_positive_mass"]) > recipient_total + or not 0.0 <= float(carrier["before_positive_share"]) <= 1.0 + or not 0.0 <= float(carrier["after_positive_share"]) <= 1.0 + or carrier["removed_rows"] > scope["effective_mutable_rows"] + or carrier["added_rows"] > scope["addition_candidate_rows"] + or carrier["disallowed_cleared_rows"] > carrier["removed_rows"] + ): + raise ValueError( + f"{boundary}: post-transfer calibration carrier relationships are invalid." + ) + + capacity = carrier.get("capacity") + selection = carrier.get("selection") + if spec.carrier_mode == "preserve_recipient": + if ( + capacity is not None + or selection is not None + or carrier["capacity_limited"] is not False + or any(carrier[key] != 0 for key in required_carrier_counts) + or not np.isclose( + float(carrier["after_positive_mass"]), + float(carrier["before_positive_mass"]), + rtol=1e-12, + atol=1e-12, + ) + ): + raise ValueError( + f"{boundary}: preserve-recipient carrier proof is invalid." + ) + else: + capacity_count_keys = { + "fixed_positive_rows", + "allowed_positive_rows_before", + "addition_candidate_rows", + } + capacity_mass_keys = { + "fixed_positive_mass", + "allowed_positive_mass_before", + "addition_candidate_mass", + "minimum_attainable_mass", + "maximum_attainable_mass", + } + expected_capacity_keys = ( + capacity_count_keys + | capacity_mass_keys + | { + "target_within_attainable_interval", + "capacity_boundary_saturated", + } + ) + selection_count_keys = { + "selected_rows", + "candidate_rows", + "chosen_prefix_rows", + } + selection_mass_keys = { + "base_positive_mass", + "selected_mass", + "requested_prefix_mass", + "candidate_mass", + "chosen_prefix_mass", + "lower_prefix_mass", + "upper_prefix_mass", + } + expected_selection_keys = ( + selection_count_keys | selection_mass_keys | {"action"} + ) + if ( + not isinstance(capacity, Mapping) + or set(capacity) != expected_capacity_keys + or not isinstance(selection, Mapping) + or set(selection) != expected_selection_keys + or any( + not isinstance(capacity.get(key), int) + or isinstance(capacity.get(key), bool) + or capacity[key] < 0 + for key in capacity_count_keys + ) + or any( + not _is_finite_number(capacity.get(key)) or capacity[key] < 0.0 + for key in capacity_mass_keys + ) + or not isinstance(capacity.get("target_within_attainable_interval"), bool) + or capacity.get("capacity_boundary_saturated") is not True + or any( + not isinstance(selection.get(key), int) + or isinstance(selection.get(key), bool) + or selection[key] < 0 + for key in selection_count_keys + ) + or any( + not _is_finite_number(selection.get(key)) or selection[key] < 0.0 + for key in selection_mass_keys + ) + or selection.get("action") + not in {"retain_positive_prefix", "add_zero_prefix"} + ): + raise ValueError( + f"{boundary}: match-reference carrier capacity proof is invalid." + ) + fixed_mass = float(capacity["fixed_positive_mass"]) + allowed_mass = float(capacity["allowed_positive_mass_before"]) + addition_mass = float(capacity["addition_candidate_mass"]) + minimum = float(capacity["minimum_attainable_mass"]) + maximum = float(capacity["maximum_attainable_mass"]) + target = float(carrier["target_positive_mass"]) + expected_limited = target < minimum or target > maximum + action = selection["action"] + desired_mutable_mass = max(0.0, target - fixed_mass) + expected_action = ( + "retain_positive_prefix" + if desired_mutable_mass <= allowed_mass + else "add_zero_prefix" + ) + expected_candidate_rows = ( + capacity["allowed_positive_rows_before"] + if action == "retain_positive_prefix" + else capacity["addition_candidate_rows"] + ) + expected_candidate_mass = ( + allowed_mass if action == "retain_positive_prefix" else addition_mass + ) + expected_base = ( + fixed_mass + if action == "retain_positive_prefix" + else (fixed_mass + allowed_mass) + ) + expected_requested = ( + max(0.0, target - fixed_mass) + if action == "retain_positive_prefix" + else max(0.0, target - fixed_mass - allowed_mass) + ) + lower = float(selection["lower_prefix_mass"]) + upper = float(selection["upper_prefix_mass"]) + expected_chosen = ( + lower + if abs(lower - expected_requested) <= abs(upper - expected_requested) + else upper + ) + after_mass = float(carrier["after_positive_mass"]) + boundary_mass = minimum if target < minimum else maximum + if ( + not np.isclose(minimum, fixed_mass, rtol=1e-12, atol=1e-12) + or action != expected_action + or capacity["fixed_positive_rows"] + > scope["recipient_rows"] - scope["effective_mutable_rows"] + or capacity["allowed_positive_rows_before"] > scope["allowed_carrier_rows"] + or capacity["addition_candidate_rows"] > scope["addition_candidate_rows"] + or sum(capacity[key] for key in capacity_count_keys) + > scope["recipient_rows"] + or fixed_mass > recipient_total + or allowed_mass > recipient_total + or addition_mass > recipient_total + or not np.isclose( + maximum, + fixed_mass + allowed_mass + addition_mass, + rtol=1e-12, + atol=1e-12, + ) + or maximum > recipient_total + or carrier["capacity_limited"] is not expected_limited + or capacity["target_within_attainable_interval"] is expected_limited + or selection["candidate_rows"] != expected_candidate_rows + or selection["selected_rows"] != selection["chosen_prefix_rows"] + or selection["selected_rows"] > selection["candidate_rows"] + or not np.isclose( + float(selection["candidate_mass"]), + expected_candidate_mass, + rtol=1e-12, + atol=1e-12, + ) + or not np.isclose( + float(selection["base_positive_mass"]), + expected_base, + rtol=1e-12, + atol=1e-12, + ) + or not np.isclose( + float(selection["requested_prefix_mass"]), + expected_requested, + rtol=1e-12, + atol=1e-12, + ) + or not np.isclose( + float(selection["selected_mass"]), + float(selection["chosen_prefix_mass"]), + rtol=1e-12, + atol=1e-12, + ) + or not np.isclose( + float(selection["chosen_prefix_mass"]), + expected_chosen, + rtol=1e-12, + atol=1e-12, + ) + or not (0.0 <= lower <= upper <= expected_candidate_mass) + or not np.isclose( + after_mass, + expected_base + float(selection["selected_mass"]), + rtol=1e-12, + atol=1e-12, + ) + or ( + expected_limited + and not np.isclose( + after_mass, + boundary_mass, + rtol=1e-12, + atol=1e-12, + ) + ) + ): + raise ValueError( + f"{boundary}: match-reference carrier capacity relationships " + "are invalid." + ) + + amount = receipt.get("amount") + expected_amount_keys = { + "quantiles", + "reference_quantiles", + "recipient_before_quantiles", + "recipient_after_quantiles", + "qed_before", + "qed_after", + "mapped_rows", + "anchor_rows", + "exact_anchor_count", + "anchor_conflicts", + "unanchored_quantiles", + "donor_support_violations", + "status", + } + if not isinstance(amount, Mapping) or set(amount) != expected_amount_keys: + raise ValueError( + f"{boundary}: post-transfer calibration amount schema is invalid." + ) + reference_quantiles = amount.get("reference_quantiles") + before_quantiles = amount.get("recipient_before_quantiles") + after_quantiles = amount.get("recipient_after_quantiles") + quantile_vectors = (reference_quantiles, after_quantiles) + if ( + amount.get("quantiles") != list(_QUANTILES) + or any( + not isinstance(vector, list) + or len(vector) != len(_QUANTILES) + or any(not _is_finite_number(value) or value <= 0.0 for value in vector) + for vector in quantile_vectors + ) + or ( + before_quantiles is not None + and ( + not isinstance(before_quantiles, list) + or len(before_quantiles) != len(_QUANTILES) + or any( + not _is_finite_number(value) or value <= 0.0 + for value in before_quantiles + ) + ) + ) + or not isinstance(amount.get("mapped_rows"), int) + or isinstance(amount.get("mapped_rows"), bool) + or amount["mapped_rows"] < 0 + or amount["mapped_rows"] > scope["effective_mutable_rows"] + or not isinstance(amount.get("exact_anchor_count"), int) + or isinstance(amount.get("exact_anchor_count"), bool) + or not 0 <= amount["exact_anchor_count"] <= len(_QUANTILES) + or not isinstance(amount.get("donor_support_violations"), int) + or isinstance(amount.get("donor_support_violations"), bool) + or amount["donor_support_violations"] != 0 + or not isinstance(amount.get("anchor_rows"), list) + or not isinstance(amount.get("anchor_conflicts"), list) + or not isinstance(amount.get("unanchored_quantiles"), list) + ): + raise ValueError( + f"{boundary}: post-transfer calibration donor support is invalid." + ) + for label, observed, expected in ( + ( + "qed_before", + amount.get("qed_before"), + _qed( + np.asarray(reference_quantiles, dtype=np.float64), + ( + None + if before_quantiles is None + else np.asarray(before_quantiles, dtype=np.float64) + ), + ), + ), + ( + "qed_after", + amount.get("qed_after"), + _qed( + np.asarray(reference_quantiles, dtype=np.float64), + np.asarray(after_quantiles, dtype=np.float64), + ), + ), + ): + if expected is None: + valid = observed is None + else: + valid = _is_finite_number(observed) and np.isclose( + float(observed), expected, rtol=1e-12, atol=1e-12 + ) + if not valid: + raise ValueError( + f"{boundary}: post-transfer calibration {label} is invalid." + ) + unanchored = amount["unanchored_quantiles"] + conflicts = amount["anchor_conflicts"] + expected_status = ( + "exact" if not conflicts and not unanchored else "infeasible_exact_anchors" + ) + anchor_keys = { + "quantile", + "entity_id", + "recipient_rank", + "reference_value", + "mutable", + } + conflict_keys = { + "reason", + "recipient_rank", + "earlier_quantile", + "earlier_reference_value", + "quantile", + "reference_value", + } + if ( + len(amount["anchor_rows"]) != len(_QUANTILES) + or any( + not isinstance(row, Mapping) + or set(row) != anchor_keys + or row.get("quantile") != probability + or not isinstance(row.get("recipient_rank"), int) + or isinstance(row.get("recipient_rank"), bool) + or row["recipient_rank"] < 0 + or row["recipient_rank"] >= scope["recipient_rows"] + or row.get("entity_id") is None + or not _is_finite_number(row.get("reference_value")) + or row["reference_value"] <= 0.0 + or not np.isclose( + float(row["reference_value"]), + float(reference_quantiles[index]), + rtol=1e-12, + atol=1e-12, + ) + or not isinstance(row.get("mutable"), bool) + for index, (row, probability) in enumerate( + zip(amount["anchor_rows"], _QUANTILES, strict=True) + ) + ) + or any( + not isinstance(conflict, Mapping) + or set(conflict) != conflict_keys + or conflict.get("reason") + not in { + "distinct_reference_values_share_recipient_row", + "recipient_anchor_row_is_immutable", + } + or not isinstance(conflict.get("recipient_rank"), int) + or isinstance(conflict.get("recipient_rank"), bool) + or conflict["recipient_rank"] < 0 + or conflict["recipient_rank"] >= scope["recipient_rows"] + or not _is_finite_number(conflict.get("quantile")) + or conflict["quantile"] not in _QUANTILES + or ( + conflict.get("earlier_quantile") is not None + and ( + not _is_finite_number(conflict["earlier_quantile"]) + or conflict["earlier_quantile"] not in _QUANTILES + ) + ) + or not _is_finite_number(conflict.get("reference_value")) + or conflict["reference_value"] <= 0.0 + or ( + conflict.get("earlier_reference_value") is not None + and ( + not _is_finite_number(conflict["earlier_reference_value"]) + or conflict["earlier_reference_value"] <= 0.0 + ) + ) + for conflict in conflicts + ) + or any( + not _is_finite_number(value) or value not in _QUANTILES + for value in unanchored + ) + or len({float(value) for value in unanchored}) != len(unanchored) + or amount["exact_anchor_count"] != len(_QUANTILES) - len(unanchored) + or amount.get("status") != expected_status + ): + raise ValueError( + f"{boundary}: post-transfer calibration anchor evidence is invalid." + ) + if require_exact_anchors and ( + invariants.get("exact_quantile_anchors") is not True + or amount.get("status") != "exact" + or amount.get("exact_anchor_count") != len(_QUANTILES) + or amount.get("anchor_conflicts") != [] + or amount.get("unanchored_quantiles") != [] + ): + raise ValueError( + f"{boundary}: post-transfer calibration exact quantile anchors are " + "infeasible." + ) + + +def apply_post_transfer_calibration( + frame: Frame, + *, + entity: str, + target: str, + reference_rows: object, + recipient_rows: object, + mutable_rows: object, + allowed_carrier_rows: object | None = None, + addition_candidate_rows: object | None = None, + family: str | None = None, +) -> PostTransferCalibrationFrameResult: + """Apply one calibration using Frame-resolved weights and entity IDs.""" + + if not isinstance(frame, Frame): + raise TypeError(f"frame must be a Frame, got {type(frame).__name__}.") + spec = ( + post_transfer_calibration_spec_for_target(entity=entity, target=target) + if family is None + else post_transfer_calibration_spec( + entity=entity, + family=family, + target=target, + ) + ) + table = frame.table(entity) + if target not in table: + raise ValueError( + f"Post-transfer calibration target {entity}/{target} is absent." + ) + id_column = frame.schema.entity_id_column(entity) + resolved_weights = frame.resolve_weights(entity) + result = calibrate_post_transfer_values( + table[target].to_numpy(copy=True), + resolved_weights.values, + table[id_column].to_numpy(copy=False), + spec=spec, + reference_rows=reference_rows, + recipient_rows=recipient_rows, + mutable_rows=mutable_rows, + allowed_carrier_rows=allowed_carrier_rows, + addition_candidate_rows=addition_candidate_rows, + ) + mutable_mask = _aligned_bool_mask( + mutable_rows, + size=len(table), + label="mutable_rows", + ) + reference_mask = _aligned_bool_mask( + reference_rows, + size=len(table), + label="reference_rows", + ) + recipient_mask = _aligned_bool_mask( + recipient_rows, + size=len(table), + label="recipient_rows", + ) + allowed_mask = ( + mutable_mask + if allowed_carrier_rows is None + else _aligned_bool_mask( + allowed_carrier_rows, + size=len(table), + label="allowed_carrier_rows", + ) + ) + addition_mask = ( + allowed_mask + if addition_candidate_rows is None + else _aligned_bool_mask( + addition_candidate_rows, + size=len(table), + label="addition_candidate_rows", + ) + ) + original_numeric = _float64_values(table[target].to_numpy(copy=False), label=target) + changed_rows = mutable_mask & ( + original_numeric.view(np.uint64) != result.values.view(np.uint64) + ) + tables = {name: frame.table(name) for name in frame.entities} + calibrated = table.copy(deep=True) + calibrated.loc[changed_rows, target] = result.values[changed_rows] + tables[entity] = calibrated + tables.update({name: frame.link(name) for name in frame.links}) + output = Frame( + tables, + frame.schema, + {name: frame.weights_for(name) for name in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + receipt = { + **result.receipt, + "scope": { + **result.receipt["scope"], + "applied_changed_rows": int(changed_rows.sum()), + }, + "weights": { + **result.receipt["weights"], + "kind": resolved_weights.kind.value, + }, + } + receipt_without_hash = dict(receipt) + receipt_without_hash.pop("sha256", None) + receipt["sha256"] = _canonical_sha256(receipt_without_hash) + validate_post_transfer_calibration_receipt( + receipt, + spec=spec, + boundary=f"Frame post-transfer calibration {spec.key}", + expected_scope={ + "reference_rows_sha256": _mask_sha256(reference_mask), + "recipient_rows_sha256": _mask_sha256(recipient_mask), + "mutable_rows_sha256": _mask_sha256(mutable_mask), + "allowed_carrier_rows_sha256": _mask_sha256(allowed_mask), + "addition_candidate_rows_sha256": _mask_sha256(addition_mask), + "entity_ids_sha256": _canonical_sha256( + [ + _json_scalar(value) + for value in table[id_column].to_numpy(copy=False).tolist() + ] + ), + "input_values_sha256": _values_sha256(original_numeric), + "output_values_sha256": _values_sha256(result.values), + }, + expected_weights_sha256=_values_sha256( + _numeric_values(resolved_weights.values, label="weights") + ), + ) + return PostTransferCalibrationFrameResult(frame=output, receipt=receipt) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py index cc49b0e6..7a5fdd9a 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/sipp_vehicles.py @@ -401,7 +401,7 @@ def load_sipp_2023_vehicle_donor( delimiter="|", usecols=list(SIPP_VEHICLE_SOURCE_COLUMNS), chunksize=int(chunksize), - low_memory=False, + low_memory=True, ) for chunk in reader: month = pd.to_numeric(chunk["MONTHCODE"], errors="coerce") diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py index 6b4ffc1f..51d70acf 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/stacked_spine.py @@ -44,7 +44,7 @@ import sys from collections import Counter from collections.abc import Callable, Mapping, Sequence -from dataclasses import asdict, dataclass, field +from dataclasses import asdict, dataclass, field, replace from importlib.resources import files from pathlib import Path from types import MappingProxyType @@ -80,6 +80,9 @@ ) from microcosm.build.serialization_dtypes import canonicalize_table_string_dtypes from microcosm.build.source_manifest import load_source_manifest +from microcosm.build.us_runtime import ( + post_transfer_calibration as post_transfer_calibration_runtime, +) from microcosm.build.us_runtime.acs_income_universe import ( ACS_PUMS_EARNINGS_SOURCE_COLUMNS, AcsPumsEarningsUniverseApplication, @@ -90,6 +93,7 @@ from microcosm.build.us_runtime.acs_transfer import ( ASEC_PUF_DONOR_SPINE, DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, + AcsImputedInput, AcsTransferResult, AcsTransferTargetBank, TargetFamilies, @@ -125,6 +129,12 @@ FORMULA_OWNED_SOURCE_COLUMNS, PRE_ASSEMBLY_OPERATOR_OUTPUT_FAMILIES, ) +from microcosm.build.us_runtime.post_transfer_calibration import ( + PostTransferCalibrationSpec, + apply_post_transfer_calibration, + post_transfer_calibration_policy_identity, + validate_post_transfer_calibration_receipt, +) from microcosm.build.us_runtime.puf_aggregate_records import ( PufAggregateDisaggregationSpec, ) @@ -207,7 +217,8 @@ US_LATE_TRANSFER_TARGET_BANK_INPUT, us_late_producer_schedule_receipt, ) -from microcosm.frame import US_SCHEMA, Frame +from microcosm.fit import Regime +from microcosm.frame import US_SCHEMA, Frame, WeightKind __all__ = [ "ACS_STACKED_SUPPORT_CHANNEL", @@ -256,6 +267,7 @@ "us_puf_s_corp_universe_zero_rule_identity", "validate_stacked_late_producer_receipt", "validate_stacked_late_producer_transition_authority", + "validate_stacked_gap_fill_receipt", "validate_stacked_post_puf_transfer_receipt", "validate_stacked_spine_frame", ] @@ -1688,11 +1700,11 @@ def thaw(item: object) -> object: _GAP_FILL_ASEC_HOUSING_TO_ACS = "asec_housing_to_acs" _GAP_FILL_HOUSING_FAMILY = "housing" _STACKED_AUTHORITY_ID = "us_stacked_spine_authority" -# v10 binds the primary-PUF whole-pool output-universe declaration. v9 bound -# the content-hashed execution/transition-authority schema in addition to the -# import-validated producer/input DAG. Version 8 named the graph but did not -# authenticate its live input/output transition. -_STACKED_AUTHORITY_VERSION = 10 +# v11 binds the post-transfer two-part calibration policy. v10 bound the +# primary-PUF whole-pool output-universe declaration. v9 bound the +# content-hashed execution/transition-authority schema in addition to the +# import-validated producer/input DAG. +_STACKED_AUTHORITY_VERSION = 11 _CANONICAL_AUTHORITY_FORM = "CANONICAL" _NONCANONICAL_AUTHORITY_FORM = "NON-CANONICAL" _PRE_CLONE_PREPARATION_STAGE = "prepare_multispine_source_inputs_for_clone" @@ -1949,6 +1961,7 @@ class _StackedAuthority: support_profile: _BatterySupportProfile puf_capital_gains_tail_support_contract: Mapping[str, object] late_producer_schedule: Mapping[str, object] + post_transfer_calibration: Mapping[str, object] declared_component_sha256: Mapping[str, str] declared_sha256: str declared_form: str @@ -2015,6 +2028,15 @@ def __post_init__(self) -> None: "late_producer_schedule", _freeze_authority_payload(self.late_producer_schedule), ) + if not isinstance(self.post_transfer_calibration, Mapping): + raise TypeError( + "Stacked authority post-transfer calibration policy must be a mapping." + ) + object.__setattr__( + self, + "post_transfer_calibration", + _freeze_authority_payload(self.post_transfer_calibration), + ) component_digests = dict(self.declared_component_sha256) if set(component_digests) != { "gap_fill_plan", @@ -2025,6 +2047,7 @@ def __post_init__(self) -> None: "support_profile", "puf_capital_gains_tail_support_contract", "late_producer_schedule", + "post_transfer_calibration", }: raise ValueError( "Stacked authority must carry every component's declared digest." @@ -2320,6 +2343,7 @@ def _authority_component_payloads( support_profile: _BatterySupportProfile, puf_capital_gains_tail_support_contract: Mapping[str, object], late_producer_schedule: Mapping[str, object], + post_transfer_calibration: Mapping[str, object], ) -> dict[str, object]: return { "gap_fill_plan": _plan_payload(gap_fill_plan), @@ -2343,6 +2367,7 @@ def _authority_component_payloads( puf_capital_gains_tail_support_contract ), "late_producer_schedule": _json_ready(late_producer_schedule), + "post_transfer_calibration": _json_ready(post_transfer_calibration), } @@ -2373,6 +2398,7 @@ def _authority_live_digests( authority.puf_capital_gains_tail_support_contract ), late_producer_schedule=authority.late_producer_schedule, + post_transfer_calibration=authority.post_transfer_calibration, ) component_digests = { name: _canonical_sha256(payload) for name, payload in payloads.items() @@ -2401,6 +2427,7 @@ def _make_stacked_authority( declared_form: str, puf_capital_gains_tail_support_contract: Mapping[str, object] | None = None, late_producer_schedule: Mapping[str, object] | None = None, + post_transfer_calibration: Mapping[str, object] | None = None, joint_metric_registry: Mapping[tuple[str, str, tuple[str, ...], int], str] | None = None, declared_component_sha256: Mapping[str, str] | None = None, @@ -2433,6 +2460,13 @@ def _make_stacked_authority( ) if not isinstance(frozen_late_producer_schedule, Mapping): raise TypeError("Late producer schedule must be a mapping.") + frozen_post_transfer_calibration = _freeze_authority_payload( + post_transfer_calibration_policy_identity() + if post_transfer_calibration is None + else post_transfer_calibration + ) + if not isinstance(frozen_post_transfer_calibration, Mapping): + raise TypeError("Post-transfer calibration policy must be a mapping.") component_payloads = _authority_component_payloads( gap_fill_plan=frozen_plan, post_puf_transfer_surface=frozen_post_puf_surface, @@ -2444,6 +2478,7 @@ def _make_stacked_authority( support_profile=support_profile, puf_capital_gains_tail_support_contract=frozen_tail_support_contract, late_producer_schedule=frozen_late_producer_schedule, + post_transfer_calibration=frozen_post_transfer_calibration, ) live_components = { name: _canonical_sha256(payload) for name, payload in component_payloads.items() @@ -2468,6 +2503,7 @@ def _make_stacked_authority( support_profile=support_profile, puf_capital_gains_tail_support_contract=frozen_tail_support_contract, late_producer_schedule=frozen_late_producer_schedule, + post_transfer_calibration=frozen_post_transfer_calibration, declared_component_sha256=( live_components if declared_component_sha256 is None @@ -3112,6 +3148,7 @@ def _production_stacked_authority( ), ) -> _StackedAuthority: live_late_producer_schedule = us_late_producer_schedule_receipt() + live_post_transfer_calibration = post_transfer_calibration_policy_identity() identity = ( _STACKED_GAP_FILL_PLAN is _canonical_plan and _STACKED_GAP_FILL_SURFACE is _canonical_gap_surface @@ -3126,6 +3163,8 @@ def _production_stacked_authority( and _BATTERY_SUPPORT_PROFILE is _canonical_profile and _json_ready(live_late_producer_schedule) == _json_ready(_canonical_authority.late_producer_schedule) + and _json_ready(live_post_transfer_calibration) + == _json_ready(_canonical_authority.post_transfer_calibration) ) if identity: return _canonical_authority @@ -3141,6 +3180,7 @@ def _production_stacked_authority( joint_metric_registry=_BATTERY_JOINT_METRIC_REGISTRY, support_profile=_BATTERY_SUPPORT_PROFILE, late_producer_schedule=live_late_producer_schedule, + post_transfer_calibration=live_post_transfer_calibration, declared_form=_CANONICAL_AUTHORITY_FORM, declared_component_sha256=_canonical_authority.declared_component_sha256, declared_sha256=_canonical_authority.declared_sha256, @@ -3451,6 +3491,15 @@ def _authority_receipt( "producer_count": authority.late_producer_schedule.get("producer_count"), "digest_matches_declared": component_integrity["late_producer_schedule"], }, + "post_transfer_calibration": { + "identity": _json_ready(authority.post_transfer_calibration), + "sha256": live_components["post_transfer_calibration"], + "declared_sha256": authority.declared_component_sha256[ + "post_transfer_calibration" + ], + "target_count": len(authority.post_transfer_calibration.get("targets", ())), + "digest_matches_declared": component_integrity["post_transfer_calibration"], + }, } return { "authority_id": authority.authority_id, @@ -3603,6 +3652,7 @@ def _authority_validation_failures( "PUF capital-gains-tail support contract", ), ("late_producer_schedule", "late producer schedule"), + ("post_transfer_calibration", "post-transfer calibration policy"), ): component = receipt["components"][name] if not component["digest_matches_declared"]: @@ -3696,10 +3746,914 @@ def _validate_production_authority_receipt( ) +def _post_transfer_live_output_diagnostics( + frame: Frame, + *, + entity: str, + target: str, + reference_rows: np.ndarray, + recipient_rows: np.ndarray, +) -> dict[str, object]: + """Independently recompute terminal carrier and amount diagnostics.""" + + table = frame.table(entity) + values = table[target].to_numpy(copy=False) + if values.dtype != np.dtype(np.float64): + raise ValueError( + f"{entity}/{target}: live calibration output must be exact float64." + ) + resolved_weights = frame.resolve_weights(entity) + weights = resolved_weights.values + reference_total = float(weights[reference_rows].sum()) + recipient_total = float(weights[recipient_rows].sum()) + reference_positive_mass = float(weights[reference_rows & (values > 0.0)].sum()) + after_positive_mass = float(weights[recipient_rows & (values > 0.0)].sum()) + reference_quantiles = post_transfer_calibration_runtime._weighted_inverse_quantiles( + values[reference_rows], + weights[reference_rows], + ) + recipient_quantiles = post_transfer_calibration_runtime._weighted_inverse_quantiles( + values[recipient_rows], + weights[recipient_rows], + ) + if reference_quantiles is None or recipient_quantiles is None: + raise ValueError( + f"{entity}/{target}: live calibration output lacks positive " + "reference or recipient quantile support." + ) + + reference_share = reference_positive_mass / reference_total + target_positive_mass = reference_share * recipient_total + after_share = after_positive_mass / recipient_total + residual = after_positive_mass - target_positive_mass + return { + "weights": { + "sha256": _post_transfer_float64_sha256( + weights, + boundary=f"{entity}/{target} live weights", + ), + "kind": resolved_weights.kind.value, + "reference_total": reference_total, + "recipient_total": recipient_total, + }, + "carrier": { + "reference_positive_mass": reference_positive_mass, + "reference_positive_share": reference_share, + "target_positive_mass": target_positive_mass, + "after_positive_mass": after_positive_mass, + "after_positive_share": after_share, + "residual_after_minus_target": residual, + "absolute_residual": abs(residual), + }, + "amount": { + "reference_quantiles": [float(value) for value in reference_quantiles], + "recipient_after_quantiles": [ + float(value) for value in recipient_quantiles + ], + "qed_after": post_transfer_calibration_runtime._qed( + reference_quantiles, + recipient_quantiles, + ), + }, + } + + +def _post_transfer_live_scope_binding( + frame: Frame, + *, + entity: str, + target: str, + reference_rows: np.ndarray, + recipient_rows: np.ndarray, +) -> dict[str, object]: + """Reconstruct the receipt scope fields available from a final frame.""" + + table = frame.table(entity) + ids = table[frame.schema.entity_id_column(entity)].to_numpy(copy=False) + values = table[target].to_numpy(copy=False) + return { + "rows": len(table), + "reference_rows": int(reference_rows.sum()), + "recipient_rows": int(recipient_rows.sum()), + "reference_rows_sha256": _post_transfer_mask_sha256(reference_rows), + "recipient_rows_sha256": _post_transfer_mask_sha256(recipient_rows), + "entity_ids_sha256": _post_transfer_entity_ids_sha256(ids), + "output_values_sha256": _post_transfer_float64_sha256( + values, + boundary=f"{entity}/{target} live output", + ), + } + + +def _validate_post_transfer_generation_context_copies( + owner_receipt: Mapping[str, object], + *, + boundary: str, +) -> None: + """Require duplicated generation-context claims to agree structurally.""" + + calibration = owner_receipt.get("calibration") + context_binding = owner_receipt.get("context_binding") + calibration_scope = ( + calibration.get("scope") if isinstance(calibration, Mapping) else None + ) + context_scope = ( + context_binding.get("scope") if isinstance(context_binding, Mapping) else None + ) + calibration_weights = ( + calibration.get("weights") if isinstance(calibration, Mapping) else None + ) + if ( + not isinstance(calibration_scope, Mapping) + or not isinstance(context_scope, Mapping) + or not isinstance(calibration_weights, Mapping) + or not isinstance(context_binding, Mapping) + or any( + calibration_scope.get(key) != value for key, value in context_scope.items() + ) + or calibration_weights.get("sha256") != context_binding.get("weights_sha256") + ): + raise ValueError( + f"{boundary}: duplicated post-transfer generation context is inconsistent." + ) + + +def _validate_post_transfer_live_output_diagnostics( + calibration: Mapping[str, object], + *, + expected: Mapping[str, object], + boundary: str, +) -> None: + """Bind terminally replayable diagnostics to live final-frame values.""" + + mismatched: list[str] = [] + for section_name in ("weights", "carrier"): + observed_section = calibration.get(section_name) + expected_section = expected[section_name] + assert isinstance(expected_section, Mapping) + if not isinstance(observed_section, Mapping): + mismatched.append(section_name) + continue + mismatched.extend( + f"{section_name}.{key}" + for key, value in expected_section.items() + if observed_section.get(key) != value + ) + + observed_amount = calibration.get("amount") + expected_amount = expected["amount"] + assert isinstance(expected_amount, Mapping) + if not isinstance(observed_amount, Mapping): + mismatched.append("amount") + else: + mismatched.extend( + f"amount.{key}" + for key in ( + "reference_quantiles", + "recipient_after_quantiles", + "qed_after", + ) + if observed_amount.get(key) != expected_amount[key] + ) + if mismatched: + raise ValueError( + f"{boundary}: calibration diagnostics do not match the live output " + f"frame for {mismatched}." + ) + + +def _validate_post_transfer_live_output_context( + frame: Frame, + *, + owner_receipt: Mapping[str, object], + spec: PostTransferCalibrationSpec, + boundary: str, +) -> None: + """Replay all receipt fields reconstructible from the supplied final frame. + + Mutable masks, input hashes, before-state diagnostics, transition counts, + and byte-preservation claims need the pre-calibration frame. The receipt's + verification contract classifies those as generation-transition evidence; + they are bound by the enclosing execution authority, not replayed here. + """ + + context_binding = owner_receipt.get("context_binding") + live_output = ( + context_binding.get("live_output") + if isinstance(context_binding, Mapping) + else None + ) + if not isinstance(live_output, Mapping): + raise ValueError(f"{boundary}: calibration context binding is absent.") + entity = spec.entity + target = spec.target + table = frame.table(entity) + channel = table[support_channel_column(entity)].astype(str) + clone_index = pd.to_numeric( + table[support_clone_index_column(entity)], + errors="raise", + ) + reference_rows = ( + channel.eq(BASE_ASEC_SUPPORT_CHANNEL) & clone_index.eq(0) + ).to_numpy(dtype=bool) + recipient_rows = ( + channel.eq(ACS_STACKED_SUPPORT_CHANNEL) & clone_index.eq(0) + ).to_numpy(dtype=bool) + expected_scope = _post_transfer_live_scope_binding( + frame, + entity=entity, + target=target, + reference_rows=reference_rows, + recipient_rows=recipient_rows, + ) + calibration = owner_receipt.get("calibration") + calibration_scope = ( + calibration.get("scope") if isinstance(calibration, Mapping) else None + ) + context_scope = ( + context_binding.get("scope") if isinstance(context_binding, Mapping) else None + ) + scope_mismatches: list[str] = [] + for label, observed in ( + ("calibration.scope", calibration_scope), + ("context_binding.scope", context_scope), + ): + if not isinstance(observed, Mapping): + scope_mismatches.append(label) + continue + scope_mismatches.extend( + f"{label}.{key}" + for key, value in expected_scope.items() + if observed.get(key) != value + ) + scope_mismatches.extend( + f"owner_receipt.{key}" + for key in ("reference_rows", "recipient_rows") + if owner_receipt.get(key) != expected_scope[key] + ) + if scope_mismatches: + raise ValueError( + f"{boundary}: calibration scope does not match the live output " + f"frame for {scope_mismatches}." + ) + expected = _post_transfer_selected_output_binding( + frame, + entity=entity, + target=target, + reference_rows=reference_rows, + recipient_rows=recipient_rows, + ) + mismatched = [ + key for key, value in expected.items() if live_output.get(key) != value + ] + if mismatched: + raise ValueError( + f"{boundary}: calibration evidence does not match the live output " + f"frame for {mismatched}." + ) + if not isinstance(calibration, Mapping): + raise ValueError(f"{boundary}: calibration receipt is absent.") + _validate_post_transfer_live_output_diagnostics( + calibration, + expected=_post_transfer_live_output_diagnostics( + frame, + entity=entity, + target=target, + reference_rows=reference_rows, + recipient_rows=recipient_rows, + ), + boundary=boundary, + ) + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + positive = recipient_rows & pd.to_numeric( + table[target], + errors="raise", + ).gt(0.0).to_numpy(dtype=bool) + qualifying = acs_transfer_runtime.acs_adult_care_qualifying_rows( + table + ).to_numpy(dtype=bool) + units = table[frame.schema.membership_column("tax_unit")] + carrier_counts = ( + pd.Series(positive, index=table.index) + .groupby( + units, + dropna=False, + ) + .sum() + ) + if ( + (positive & ~qualifying).any() + or units.loc[positive].isna().any() + or carrier_counts.gt(1).any() + ): + raise ValueError( + f"{boundary}: live adult-care carriers violate qualifying-person " + "or one-carrier-per-tax-unit structure." + ) + elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": + positive_weeks = recipient_rows & pd.to_numeric( + table[target], + errors="raise", + ).gt(0.0).to_numpy(dtype=bool) + positive_unemployment = ( + pd.to_numeric( + table["unemployment_compensation"], + errors="raise", + ) + .gt(0.0) + .to_numpy(dtype=bool) + ) + if (positive_weeks & ~positive_unemployment).any(): + raise ValueError( + f"{boundary}: live positive weeks-unemployed carriers lack " + "positive unemployment compensation." + ) + + +def _validate_post_transfer_live_output_binding( + owner_receipt: Mapping[str, object], + *, + boundary: str, +) -> None: + """Require complete syntactic terminal binding even without a live frame.""" + + context_binding = owner_receipt.get("context_binding") + live_output = ( + context_binding.get("live_output") + if isinstance(context_binding, Mapping) + else None + ) + digest_keys = { + "reference_entity_ids_sha256", + "recipient_entity_ids_sha256", + "reference_output_values_sha256", + "recipient_output_values_sha256", + "reference_weights_sha256", + "recipient_weights_sha256", + } + if not isinstance(live_output, Mapping) or not ( + digest_keys | {"reference_rows", "recipient_rows"} + ).issubset(live_output): + raise ValueError(f"{boundary}: calibration live-output binding is absent.") + if any( + not isinstance(live_output.get(key), int) + or isinstance(live_output.get(key), bool) + or live_output[key] < 0 + for key in ("reference_rows", "recipient_rows") + ) or any( + not isinstance(live_output.get(key), str) + or len(live_output[key]) != 64 + or any(character not in "0123456789abcdef" for character in live_output[key]) + for key in digest_keys + ): + raise ValueError(f"{boundary}: calibration live-output binding is invalid.") + + +_ACS_QRF_REGIMES = frozenset( + { + Regime.THREE_SIGN, + Regime.ZERO_INFLATED_POSITIVE, + Regime.ZERO_INFLATED_NEGATIVE, + Regime.SIGN_ONLY, + Regime.POSITIVE_ONLY, + Regime.NEGATIVE_ONLY, + Regime.DEGENERATE_ZERO, + } +) +_ACS_QRF_PATTERN_EVIDENCE_KIND = "acs_transfer_qrf_pattern_regimes" +_ACS_QRF_PATTERN_EVIDENCE_SCHEMA_VERSION = 1 +_ACS_QRF_WEIGHT_KINDS = frozenset(kind.value for kind in WeightKind) +_ACS_TRANSFER_ROW_COUNT_FIELDS = ( + "authorized_null_rows", + "imputed_rows", + "unmodeled_rows", + "residual_null_rows", +) + + +def _validate_acs_transfer_row_counts( + target_receipt: Mapping[str, object], + *, + boundary: str, + required: bool = False, +) -> dict[str, int]: + """Validate the legacy transfer counts independently of opt-in evidence.""" + + present = { + field for field in _ACS_TRANSFER_ROW_COUNT_FIELDS if field in target_receipt + } + if not present: + if required: + raise ValueError(f"{boundary}: ACS transfer row-count schema is invalid.") + return {} + if present != set(_ACS_TRANSFER_ROW_COUNT_FIELDS): + raise ValueError(f"{boundary}: ACS transfer row-count schema is invalid.") + counts = {field: target_receipt[field] for field in _ACS_TRANSFER_ROW_COUNT_FIELDS} + if any( + not isinstance(value, int) or isinstance(value, bool) or value < 0 + for value in counts.values() + ): + raise ValueError(f"{boundary}: ACS transfer row-count schema is invalid.") + typed_counts = {field: int(value) for field, value in counts.items()} + if ( + typed_counts["authorized_null_rows"] + != typed_counts["imputed_rows"] + typed_counts["unmodeled_rows"] + or typed_counts["residual_null_rows"] != typed_counts["unmodeled_rows"] + ): + raise ValueError(f"{boundary}: ACS transfer row-count accounting is invalid.") + return typed_counts + + +def _acs_imputed_pattern_evidence(record: AcsImputedInput) -> dict[str, object]: + """Return deterministic JSON-ready evidence for one ACS transfer record.""" + + if not isinstance(record, AcsImputedInput): + raise TypeError("ACS QRF pattern evidence requires an AcsImputedInput.") + patterns = [ + { + "name": pattern.name, + "observed_optional_predictors": list(pattern.observed_optional_predictors), + "predictors": list(pattern.predictors), + "seed": int(pattern.seed), + "weight_kind": pattern.weight_kind, + "donor_rows": int(pattern.donor_rows), + "recipient_rows": int(pattern.recipient_rows), + "target_regimes": [ + {"model_target": model_target, "regime": regime} + for model_target, regime in pattern.target_regimes + ], + } + for pattern in record.patterns + ] + payload: dict[str, object] = { + "evidence_kind": _ACS_QRF_PATTERN_EVIDENCE_KIND, + "schema_version": _ACS_QRF_PATTERN_EVIDENCE_SCHEMA_VERSION, + "record": { + "entity": record.entity, + "family": record.family, + "column": record.column, + "predictors": list(record.predictors), + "seed": int(record.seed), + "weight_kind": record.weight_kind, + "imputed_recipient_rows": int(record.imputed_recipient_rows), + "unmodeled_recipient_rows": int(record.unmodeled_recipient_rows), + }, + "pattern_count": len(patterns), + "pattern_recipient_rows": sum( + int(pattern.recipient_rows) for pattern in record.patterns + ), + "patterns": patterns, + } + return {**payload, "sha256": _canonical_sha256(payload)} + + +def _acs_pattern_predictor_authority( + *, + entity: str, + family_targets: Sequence[str], +) -> tuple[tuple[str, ...], tuple[str, ...]]: + """Return the canonical required and optional predictor orders.""" + + if entity == "person": + required = acs_transfer_runtime.ACS_PERSON_TRANSFER_PREDICTORS + optional = acs_transfer_runtime.ACS_OPTIONAL_PERSON_TRANSFER_PREDICTORS + contract = acs_transfer_runtime.acs_transfer_execution_contract_identity( + targets=family_targets, + derive_schedule_d=False, + ) + housing = contract["housing"] + assert isinstance(housing, Mapping) + if set(family_targets).intersection(housing["targets"]): + mandatory = tuple(housing["mandatory_features"]) + required = (*required, *mandatory) + optional = tuple(item for item in optional if item not in mandatory) + return tuple(required), tuple(optional) + optional_names = acs_transfer_runtime._GROUP_OPTIONAL_NAMES + return ( + acs_transfer_runtime.ACS_GROUP_TRANSFER_PREDICTORS, + tuple( + optional_names[item] + for item in acs_transfer_runtime.ACS_OPTIONAL_PERSON_TRANSFER_PREDICTORS + ), + ) + + +def _validate_acs_imputed_pattern_evidence( + target_receipt: Mapping[str, object], + *, + expected_entity: str, + expected_family: str, + expected_target: str, + expected_family_targets: Sequence[str], + expected_regime_targets: Sequence[str] | None = None, + boundary: str, +) -> None: + """Validate independently checkable ACS QRF pattern receipt structure. + + The enclosing gap-fill manifest or late execution-chain signature binds + the reported seeds and regimes. Receipt-only validation cannot recompute + either without the transfer seed and exact encoded donor values, so this + function validates their types, vocabulary, and target placement rather + than claiming donor replay or out-of-sample verification. + """ + + evidence = target_receipt.get("qrf_pattern_evidence") + receipt_counts = _validate_acs_transfer_row_counts( + target_receipt, + boundary=boundary, + ) + claimed_record_rows = receipt_counts.get("imputed_rows", 0) + receipt_counts.get( + "unmodeled_rows", 0 + ) + if evidence is None: + if claimed_record_rows: + raise ValueError( + f"{boundary}: ACS QRF pattern evidence is absent for a target " + "with realized transfer rows." + ) + return + if not isinstance(evidence, Mapping): + raise ValueError(f"{boundary}: ACS QRF pattern evidence is not an object.") + expected_evidence_keys = { + "evidence_kind", + "schema_version", + "record", + "pattern_count", + "pattern_recipient_rows", + "patterns", + "sha256", + } + if set(evidence) != expected_evidence_keys: + raise ValueError(f"{boundary}: ACS QRF pattern evidence schema is invalid.") + unsigned = dict(evidence) + observed_sha256 = unsigned.pop("sha256") + if not isinstance(observed_sha256, str) or observed_sha256 != _canonical_sha256( + unsigned + ): + raise ValueError(f"{boundary}: ACS QRF pattern evidence SHA-256 mismatch.") + patterns = evidence.get("patterns") + record = evidence.get("record") + pattern_count = evidence.get("pattern_count") + pattern_recipient_rows = evidence.get("pattern_recipient_rows") + if ( + evidence.get("evidence_kind") != _ACS_QRF_PATTERN_EVIDENCE_KIND + or evidence.get("schema_version") != _ACS_QRF_PATTERN_EVIDENCE_SCHEMA_VERSION + or not isinstance(pattern_count, int) + or isinstance(pattern_count, bool) + or pattern_count < 0 + or not isinstance(patterns, list) + or pattern_count != len(patterns) + or not isinstance(pattern_recipient_rows, int) + or isinstance(pattern_recipient_rows, bool) + or pattern_recipient_rows < 0 + or not isinstance(record, Mapping) + or (claimed_record_rows > 0 and pattern_count == 0) + ): + raise ValueError(f"{boundary}: ACS QRF pattern evidence header is invalid.") + + expected_record_keys = { + "entity", + "family", + "column", + "predictors", + "seed", + "weight_kind", + "imputed_recipient_rows", + "unmodeled_recipient_rows", + } + if set(record) != expected_record_keys: + raise ValueError(f"{boundary}: ACS QRF pattern record schema is invalid.") + record_predictors = record.get("predictors") + record_seed = record.get("seed") + record_weight_kind = record.get("weight_kind") + if ( + record.get("entity") != expected_entity + or record.get("family") != expected_family + or record.get("column") != expected_target + or not isinstance(record_predictors, list) + or any(not isinstance(item, str) for item in record_predictors) + or len(set(record_predictors)) != len(record_predictors) + or not isinstance(record_seed, int) + or isinstance(record_seed, bool) + or not 0 <= record_seed < 2**32 + or record_weight_kind not in _ACS_QRF_WEIGHT_KINDS + or record.get("imputed_recipient_rows") != receipt_counts.get("imputed_rows") + or record.get("unmodeled_recipient_rows") + != receipt_counts.get("unmodeled_rows") + ): + raise ValueError(f"{boundary}: ACS QRF pattern record binding is invalid.") + expected_targets = acs_transfer_runtime._selected_model_target_names( + expected_family_targets, + ( + expected_family_targets + if expected_regime_targets is None + else expected_regime_targets + ), + ) + required_predictors, optional_authority = _acs_pattern_predictor_authority( + entity=expected_entity, + family_targets=expected_family_targets, + ) + expected_pattern_keys = { + "name", + "observed_optional_predictors", + "predictors", + "seed", + "weight_kind", + "donor_rows", + "recipient_rows", + "target_regimes", + } + observed_pattern_options: list[tuple[str, ...]] = [] + observed_pattern_codes: list[int] = [] + observed_pattern_recipient_rows = 0 + for index, pattern in enumerate(patterns): + if not isinstance(pattern, Mapping) or set(pattern) != expected_pattern_keys: + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} is invalid." + ) + optional = pattern.get("observed_optional_predictors") + predictors = pattern.get("predictors") + target_regimes = pattern.get("target_regimes") + if ( + not isinstance(optional, list) + or any(not isinstance(item, str) for item in optional) + or len(set(optional)) != len(optional) + or not isinstance(predictors, list) + or any(not isinstance(item, str) for item in predictors) + or len(set(predictors)) != len(predictors) + or not isinstance(pattern.get("seed"), int) + or isinstance(pattern.get("seed"), bool) + or not 0 <= pattern["seed"] < 2**32 + or pattern.get("weight_kind") != record_weight_kind + or any( + not isinstance(pattern.get(field_name), int) + or isinstance(pattern.get(field_name), bool) + or pattern[field_name] < 1 + for field_name in ("donor_rows", "recipient_rows") + ) + or not isinstance(target_regimes, list) + ): + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} metadata is invalid." + ) + optional_tuple = tuple(optional) + if optional_tuple != tuple( + item for item in optional_authority if item in optional_tuple + ) or tuple(predictors) != (*required_predictors, *optional_tuple): + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} predictor " + "order is outside canonical transfer authority." + ) + expected_name = acs_transfer_runtime._pattern_name(index, optional_tuple) + if pattern.get("name") != expected_name: + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} name is not " + "derived from its observed optional predictors." + ) + availability_code = sum( + 1 << optional_authority.index(item) for item in optional_tuple + ) + observed_pattern_options.append(optional_tuple) + observed_pattern_codes.append(availability_code) + observed_pattern_recipient_rows += pattern["recipient_rows"] + observed_targets: list[str] = [] + for target_regime in target_regimes: + if ( + not isinstance(target_regime, Mapping) + or set(target_regime) != {"model_target", "regime"} + or not isinstance(target_regime.get("model_target"), str) + or target_regime.get("regime") not in _ACS_QRF_REGIMES + ): + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} target " + "regime is invalid." + ) + observed_targets.append(target_regime["model_target"]) + if tuple(observed_targets) != expected_targets: + raise ValueError( + f"{boundary}: ACS QRF pattern evidence row {index} target order " + f"is {observed_targets}, expected {list(expected_targets)}." + ) + if ( + len(set(observed_pattern_options)) != pattern_count + or observed_pattern_codes != sorted(observed_pattern_codes) + or len(set(observed_pattern_codes)) != pattern_count + or observed_pattern_recipient_rows != pattern_recipient_rows + or receipt_counts.get("imputed_rows", 0) > pattern_recipient_rows + or ( + len(expected_family_targets) == 1 + and receipt_counts.get("imputed_rows") != pattern_recipient_rows + ) + ): + raise ValueError( + f"{boundary}: ACS QRF pattern order or recipient-row accounting is invalid." + ) + used_predictors = tuple( + predictor + for predictor in (*required_predictors, *optional_authority) + if any(predictor in pattern["predictors"] for pattern in patterns) + ) + if tuple(record_predictors) != used_predictors: + raise ValueError( + f"{boundary}: ACS QRF record predictor surface is not reconstructed " + "from its ordered patterns." + ) + + +def validate_stacked_gap_fill_receipt( + receipt: Mapping[str, object], + *, + boundary: str, + frame: Frame | None = None, +) -> None: + """Reject a canonical gap-fill proof with missing calibration evidence.""" + + if not isinstance(receipt, Mapping): + raise ValueError(f"{boundary}: stacked gap-fill receipt is absent.") + authority = receipt.get("authority") + if not isinstance(authority, Mapping): + raise ValueError(f"{boundary}: stacked gap-fill authority is absent.") + _validate_production_authority_receipt(authority, boundary=boundary) + expected_policy_sha256 = str( + _CANONICAL_STACKED_AUTHORITY.post_transfer_calibration["sha256"] + ) + early_specs = { + spec.key: spec + for spec in ( + post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + ) + if spec.stage == "early_gap_fill" + } + expected_directions = { + direction.name: direction for direction in CANONICAL_STACKED_GAP_FILL_PLAN + } + directions = receipt.get("directions") + if not isinstance(directions, Mapping) or set(directions) != set( + expected_directions + ): + raise ValueError( + f"{boundary}: stacked gap-fill direction surface is non-canonical." + ) + validated_keys: set[str] = set() + for name, direction in expected_directions.items(): + direction_receipt = directions[name] + if not isinstance(direction_receipt, Mapping): + raise ValueError( + f"{boundary}: stacked gap-fill direction {name!r} is absent." + ) + targets = direction_receipt.get("targets") + if not isinstance(targets, Mapping): + raise ValueError( + f"{boundary}: stacked gap-fill direction {name!r} has no targets." + ) + direction_target_context = { + f"{entity}/{family}/{target}": ( + entity, + family, + target, + family_targets, + ) + for entity, families in direction.target_families.items() + for family, family_targets in families.items() + for target in family_targets + } + direction_target_keys = set(direction_target_context) + if set(targets) != direction_target_keys: + raise ValueError( + f"{boundary}: stacked gap-fill direction {name!r} target " + "surface is non-canonical." + ) + expected_calibrations = { + key: early_specs[key] for key in direction_target_keys if key in early_specs + } + expected_summary = { + "policy_sha256": expected_policy_sha256, + "target_count": len(expected_calibrations), + "targets": sorted(expected_calibrations), + } + if direction_receipt.get("post_transfer_calibration") != expected_summary: + raise ValueError( + f"{boundary}: stacked gap-fill direction {name!r} has stripped " + "or misbound calibration summary evidence." + ) + validated_keys.update(expected_calibrations) + for key, target_receipt in targets.items(): + if not isinstance(target_receipt, Mapping): + raise ValueError( + f"{boundary}: stacked gap-fill target {key!r} has no receipt." + ) + _validate_acs_transfer_row_counts( + target_receipt, + boundary=f"{boundary} target {key}", + required=True, + ) + owner_receipt = target_receipt.get("post_transfer_calibration") + spec = expected_calibrations.get(key) + if spec is None: + if "qrf_pattern_evidence" in target_receipt: + raise ValueError( + f"{boundary}: undeclared ACS QRF pattern evidence is " + f"attached to {key!r}." + ) + if owner_receipt is not None: + raise ValueError( + f"{boundary}: undeclared gap-fill calibration evidence " + f"is attached to {key!r}." + ) + continue + entity, family, target, family_targets = direction_target_context[key] + expected_regime_targets = tuple( + family_target + for family_target in family_targets + if f"{entity}/{family}/{family_target}" in early_specs + ) + _validate_acs_imputed_pattern_evidence( + target_receipt, + expected_entity=entity, + expected_family=family, + expected_target=target, + expected_family_targets=family_targets, + expected_regime_targets=expected_regime_targets, + boundary=f"{boundary} target {key}", + ) + if not isinstance(owner_receipt, Mapping) or any( + owner_receipt.get(field) != expected + for field, expected in { + "stage": "early_gap_fill", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + }.items() + ): + raise ValueError( + f"{boundary}: gap-fill calibration owner selection is " + f"misbound for {key!r}." + ) + calibration = owner_receipt.get("calibration") + context_binding = owner_receipt.get("context_binding") + if ( + not isinstance(context_binding, Mapping) + or not isinstance(context_binding.get("scope"), Mapping) + or not isinstance(context_binding.get("weights_sha256"), str) + ): + raise ValueError( + f"{boundary}: gap-fill calibration context binding is " + f"absent for {key!r}." + ) + _validate_post_transfer_live_output_binding( + owner_receipt, + boundary=f"{boundary} target {key}", + ) + _validate_post_transfer_generation_context_copies( + owner_receipt, + boundary=f"{boundary} target {key}", + ) + validate_post_transfer_calibration_receipt( + calibration, + spec=spec, + boundary=f"{boundary} target {key}", + expected_policy_sha256=expected_policy_sha256, + ) + assert isinstance(calibration, Mapping) + scope = calibration["scope"] + assert isinstance(scope, Mapping) + count_keys = ("reference_rows", "recipient_rows", "mutable_rows") + if any( + not isinstance(owner_receipt.get(count_key), int) + or isinstance(owner_receipt.get(count_key), bool) + or owner_receipt[count_key] < 0 + or owner_receipt[count_key] != scope.get(count_key) + for count_key in count_keys + ): + raise ValueError( + f"{boundary}: gap-fill calibration row counts are misbound " + f"for {key!r}." + ) + if owner_receipt.get("constraint") != {"constraint": "none"}: + raise ValueError( + f"{boundary}: gap-fill calibration constraint is misbound " + f"for {key!r}." + ) + if frame is not None: + _validate_post_transfer_live_output_context( + frame, + owner_receipt=owner_receipt, + spec=spec, + boundary=f"{boundary} target {key}", + ) + if validated_keys != set(early_specs): + raise ValueError( + f"{boundary}: gap-fill calibration coverage does not match the " + "canonical early target surface." + ) + + def validate_stacked_post_puf_transfer_receipt( receipt: Mapping[str, object], *, boundary: str, + frame: Frame | None = None, ) -> None: """Reject a late-transfer receipt unless its full DAG proof is canonical.""" @@ -3712,6 +4666,19 @@ def validate_stacked_post_puf_transfer_receipt( "production manifest emission is forbidden." ) _validate_production_authority_receipt(authority, boundary=boundary) + expected_policy_sha256 = str( + _CANONICAL_STACKED_AUTHORITY.post_transfer_calibration["sha256"] + ) + late_calibration_specs = { + spec.key: spec + for spec in ( + post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + ) + if spec.stage == "late_transfer" + } + late_specs_by_target = { + (spec.entity, spec.target): spec for spec in late_calibration_specs.values() + } schedule = receipt.get("producer_schedule") expected_schedule = _json_ready(us_late_producer_schedule_receipt()) if not isinstance(schedule, Mapping) or _json_ready(schedule) != expected_schedule: @@ -3738,6 +4705,13 @@ def validate_stacked_post_puf_transfer_receipt( "canonical 19-group partition; production manifest emission is " "forbidden." ) + validated_calibration_keys: set[str] = set() + canonical_family = { + (entity, target): family + for entity, families in CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + for family, family_targets in families.items() + for target in family_targets + } for name, group in expected_groups.items(): group_receipt = groups[name] if ( @@ -3749,6 +4723,191 @@ def validate_stacked_post_puf_transfer_receipt( f"{boundary}: stacked post-PUF transfer group {name!r} is " "misbound; production manifest emission is forbidden." ) + group_targets = group_receipt.get("targets") + if not isinstance(group_targets, Mapping): + raise ValueError( + f"{boundary}: stacked post-PUF transfer group {name!r} has no " + "target receipts; production manifest emission is forbidden." + ) + expected_group_target_keys = { + f"{group.entity}/{group.family}/{target}" for target in group.targets + } + if set(group_targets) != expected_group_target_keys: + raise ValueError( + f"{boundary}: stacked post-PUF transfer group {name!r} target " + "surface is non-canonical; production manifest emission is " + "forbidden." + ) + expected_calibrations = { + f"{group.entity}/{group.family}/{target}": spec + for target in group.targets + if (spec := late_specs_by_target.get((group.entity, target))) is not None + and _stacked_calibration_family_matches( + spec, + entity=group.entity, + family=group.family, + target=target, + stage="late_transfer", + ) + } + expected_calibration_summary = { + "policy_sha256": expected_policy_sha256, + "target_count": len(expected_calibrations), + "targets": sorted(spec.key for spec in expected_calibrations.values()), + } + if group_receipt.get("post_transfer_calibration") != ( + expected_calibration_summary + ): + raise ValueError( + f"{boundary}: stacked post-PUF transfer group {name!r} has " + "stripped or misbound calibration summary evidence." + ) + validated_calibration_keys.update( + spec.key for spec in expected_calibrations.values() + ) + for target_key, target_receipt in group_targets.items(): + if not isinstance(target_receipt, Mapping): + raise ValueError( + f"{boundary}: stacked post-PUF target {target_key!r} has " + "no receipt." + ) + _validate_acs_transfer_row_counts( + target_receipt, + boundary=f"{boundary} target {target_key}", + required=True, + ) + owner_receipt = target_receipt.get("post_transfer_calibration") + spec = expected_calibrations.get(target_key) + if spec is None: + if "qrf_pattern_evidence" in target_receipt: + raise ValueError( + f"{boundary}: undeclared ACS QRF pattern evidence is " + f"attached to {target_key!r}." + ) + if owner_receipt is not None: + raise ValueError( + f"{boundary}: undeclared post-transfer calibration " + f"evidence is attached to {target_key!r}." + ) + continue + expected_regime_targets = tuple( + target + for target in group.targets + if f"{group.entity}/{group.family}/{target}" in expected_calibrations + ) + _validate_acs_imputed_pattern_evidence( + target_receipt, + expected_entity=group.entity, + expected_family=group.family, + expected_target=target_key.rsplit("/", 1)[1], + expected_family_targets=group.targets, + expected_regime_targets=expected_regime_targets, + boundary=f"{boundary} target {target_key}", + ) + if not isinstance(owner_receipt, Mapping): + raise ValueError( + f"{boundary}: post-transfer calibration evidence is " + f"absent for {target_key!r}." + ) + if any( + owner_receipt.get(key) != value + for key, value in { + "stage": "late_transfer", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + }.items() + ): + raise ValueError( + f"{boundary}: post-transfer calibration owner selection " + f"is misbound for {target_key!r}." + ) + calibration = owner_receipt.get("calibration") + context_binding = owner_receipt.get("context_binding") + if ( + not isinstance(context_binding, Mapping) + or not isinstance(context_binding.get("scope"), Mapping) + or not isinstance(context_binding.get("weights_sha256"), str) + ): + raise ValueError( + f"{boundary}: post-transfer calibration context binding " + f"is absent for {target_key!r}." + ) + _validate_post_transfer_live_output_binding( + owner_receipt, + boundary=f"{boundary} target {target_key}", + ) + _validate_post_transfer_generation_context_copies( + owner_receipt, + boundary=f"{boundary} target {target_key}", + ) + validate_post_transfer_calibration_receipt( + calibration, + spec=spec, + boundary=f"{boundary} target {target_key}", + expected_policy_sha256=expected_policy_sha256, + ) + assert isinstance(calibration, Mapping) + calibration_scope = calibration["scope"] + assert isinstance(calibration_scope, Mapping) + for count_key in ("reference_rows", "recipient_rows", "mutable_rows"): + if ( + not isinstance(owner_receipt.get(count_key), int) + or isinstance(owner_receipt.get(count_key), bool) + or owner_receipt[count_key] < 0 + or owner_receipt[count_key] != calibration_scope.get(count_key) + ): + raise ValueError( + f"{boundary}: post-transfer calibration {count_key} " + f"evidence is misbound for {target_key!r}." + ) + constraint = owner_receipt.get("constraint") + if ( + not isinstance(constraint, Mapping) + or constraint.get("constraint") != spec.special_constraint + ): + raise ValueError( + f"{boundary}: post-transfer calibration constraint is " + f"misbound for {target_key!r}." + ) + if spec.special_constraint == ("adult_care_qualifying_one_per_tax_unit"): + if ( + constraint.get("qualifying_mutable_rows") + != calibration_scope.get("allowed_carrier_rows") + or constraint.get("one_per_empty_tax_unit_addition_candidates") + != calibration_scope.get("addition_candidate_rows") + or not isinstance(owner_receipt.get("post_reconciliation"), Mapping) + or owner_receipt["post_reconciliation"].get("status") + != "verified_no_op" + ): + raise ValueError( + f"{boundary}: adult-care calibration structure evidence " + f"is invalid for {target_key!r}." + ) + elif spec.special_constraint == ( + "weeks_requires_positive_unemployment_compensation" + ) and ( + constraint.get("positive_unemployment_mutable_rows") + != calibration_scope.get("allowed_carrier_rows") + or calibration_scope.get("allowed_carrier_rows") + != calibration_scope.get("addition_candidate_rows") + ): + raise ValueError( + f"{boundary}: weeks-unemployed calibration constraint " + f"evidence is invalid for {target_key!r}." + ) + if frame is not None: + _validate_post_transfer_live_output_context( + frame, + owner_receipt=owner_receipt, + spec=spec, + boundary=f"{boundary} target {target_key}", + ) + if validated_calibration_keys != set(late_calibration_specs): + raise ValueError( + f"{boundary}: post-transfer calibration coverage does not match " + "the canonical late target surface." + ) expected_target_labels = { f"{entity}/{family}/{target}" for entity, families in CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() @@ -3762,6 +4921,23 @@ def validate_stacked_post_puf_transfer_receipt( "canonical 70-target surface; production manifest emission is " "forbidden." ) + for name, group in expected_groups.items(): + group_receipt = groups[name] + assert isinstance(group_receipt, Mapping) + group_targets = group_receipt["targets"] + assert isinstance(group_targets, Mapping) + for target in group.targets: + bounded_label = f"{group.entity}/{group.family}/{target}" + aggregate_label = ( + f"{group.entity}/{canonical_family[(group.entity, target)]}/{target}" + ) + if _json_ready(targets[aggregate_label]) != _json_ready( + group_targets[bounded_label] + ): + raise ValueError( + f"{boundary}: aggregate post-PUF target {aggregate_label!r} is not " + "reconstructed from its group receipt." + ) if any( not isinstance(target_receipt, Mapping) or target_receipt.get("residual_null_rows") != 0 @@ -6990,7 +8166,11 @@ def validate_stacked_late_producer_receipt( raise ValueError( f"{boundary}: stacked late-producer DAG transfer proof is absent." ) - validate_stacked_post_puf_transfer_receipt(transfer, boundary=boundary) + validate_stacked_post_puf_transfer_receipt( + transfer, + boundary=boundary, + frame=frame, + ) groups = transfer["groups"] assert isinstance(groups, Mapping) canonical_family = { @@ -7534,6 +8714,407 @@ def validate_structural_absence_receipt( ) +def _surface_target_triples( + surface: TargetFamilies, +) -> set[tuple[str, str, str]]: + return { + (entity, family, target) + for entity, families in surface.items() + for family, targets in families.items() + for target in targets + } + + +def _one_candidate_per_adult_care_tax_unit( + frame: Frame, + *, + mutable_rows: pd.Series, + allowed_rows: pd.Series, +) -> pd.Series: + """Choose one stable eligible zero per unit that has no current carrier.""" + + person = frame.table("person") + target = "pre_subsidy_care_expenses" + values = pd.to_numeric(person[target], errors="raise") + units = person[frame.schema.membership_column("tax_unit")] + entity_ids = person[frame.schema.entity_id_column("person")] + current_positive = values.gt(0.0) + unit_has_positive = current_positive.groupby(units).transform("any") + candidates = mutable_rows & allowed_rows & values.eq(0.0) & ~unit_has_positive + candidate_ids = entity_ids.where(candidates) + smallest_id = candidate_ids.groupby(units).transform("min") + return candidates & entity_ids.eq(smallest_id) + + +def _post_transfer_mask_sha256(mask: np.ndarray) -> str: + return hashlib.sha256( + np.ascontiguousarray(mask, dtype=bool).tobytes(order="C") + ).hexdigest() + + +def _post_transfer_float64_sha256(values: object, *, boundary: str) -> str: + array = np.asarray(values) + if array.ndim != 1 or array.dtype != np.dtype(np.float64): + raise ValueError( + f"{boundary}: post-transfer context requires an exact float64 vector." + ) + return hashlib.sha256(np.ascontiguousarray(array).tobytes(order="C")).hexdigest() + + +def _post_transfer_entity_ids_sha256(values: object) -> str: + array = np.asarray(values) + scalars = [ + value + if value is None or isinstance(value, (str, int, float, bool)) + else repr(value) + for value in array.tolist() + ] + return _canonical_sha256(scalars) + + +def _post_transfer_selected_output_binding( + frame: Frame, + *, + entity: str, + target: str, + reference_rows: np.ndarray, + recipient_rows: np.ndarray, +) -> dict[str, object]: + """Hash stable clone-0 row identities, output values, and resolved weights.""" + + table = frame.table(entity) + ids = table[frame.schema.entity_id_column(entity)].to_numpy(copy=False) + values = table[target].to_numpy(copy=False) + weights = frame.resolve_weights(entity).values + return { + "reference_rows": int(reference_rows.sum()), + "recipient_rows": int(recipient_rows.sum()), + "reference_entity_ids_sha256": _post_transfer_entity_ids_sha256( + ids[reference_rows] + ), + "recipient_entity_ids_sha256": _post_transfer_entity_ids_sha256( + ids[recipient_rows] + ), + "reference_output_values_sha256": _post_transfer_float64_sha256( + values[reference_rows], + boundary=f"{entity}/{target} reference output", + ), + "recipient_output_values_sha256": _post_transfer_float64_sha256( + values[recipient_rows], + boundary=f"{entity}/{target} recipient output", + ), + "reference_weights_sha256": _post_transfer_float64_sha256( + weights[reference_rows], + boundary=f"{entity}/{target} reference weights", + ), + "recipient_weights_sha256": _post_transfer_float64_sha256( + weights[recipient_rows], + boundary=f"{entity}/{target} recipient weights", + ), + } + + +def _post_transfer_calibration_context_binding( + input_frame: Frame, + output_frame: Frame, + *, + entity: str, + target: str, + reference_rows: np.ndarray, + recipient_rows: np.ndarray, + mutable_rows: np.ndarray, + allowed_carrier_rows: np.ndarray, + addition_candidate_rows: np.ndarray, +) -> dict[str, object]: + """Independently bind one owner selection to its live input/output vectors.""" + + input_table = input_frame.table(entity) + output_table = output_frame.table(entity) + ids = input_table[input_frame.schema.entity_id_column(entity)].to_numpy(copy=False) + weights = input_frame.resolve_weights(entity).values + return { + "scope": { + "rows": len(input_table), + "reference_rows": int(reference_rows.sum()), + "recipient_rows": int(recipient_rows.sum()), + "mutable_rows": int(mutable_rows.sum()), + "allowed_carrier_rows": int(allowed_carrier_rows.sum()), + "addition_candidate_rows": int(addition_candidate_rows.sum()), + "reference_rows_sha256": _post_transfer_mask_sha256(reference_rows), + "recipient_rows_sha256": _post_transfer_mask_sha256(recipient_rows), + "mutable_rows_sha256": _post_transfer_mask_sha256(mutable_rows), + "allowed_carrier_rows_sha256": _post_transfer_mask_sha256( + allowed_carrier_rows + ), + "addition_candidate_rows_sha256": _post_transfer_mask_sha256( + addition_candidate_rows + ), + "entity_ids_sha256": _post_transfer_entity_ids_sha256(ids), + "input_values_sha256": _post_transfer_float64_sha256( + input_table[target].to_numpy(copy=False), + boundary=f"{entity}/{target} input", + ), + "output_values_sha256": _post_transfer_float64_sha256( + output_table[target].to_numpy(copy=False), + boundary=f"{entity}/{target} output", + ), + }, + "weights_sha256": _post_transfer_float64_sha256( + weights, + boundary=f"{entity}/{target} weights", + ), + "live_output": _post_transfer_selected_output_binding( + output_frame, + entity=entity, + target=target, + reference_rows=reference_rows, + recipient_rows=recipient_rows, + ), + } + + +def _stacked_calibration_family_matches( + spec: PostTransferCalibrationSpec, + *, + entity: str, + family: str, + target: str, + stage: str, +) -> bool: + """Bind specs only to exact or authority-derived bounded families.""" + + if entity != spec.entity or target != spec.target: + return False + if family == spec.family: + return True + if stage != "late_transfer": + return False + canonical_family = { + (canonical_entity, canonical_target): canonical_family + for canonical_entity, families in ( + CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE.items() + ) + for canonical_family, targets in families.items() + for canonical_target in targets + }.get((entity, target)) + bounded_families = { + group.family + for group in CANONICAL_US_LATE_TRANSFER_GROUPS + if group.entity == entity and target in group.targets + } + return canonical_family == spec.family and family in bounded_families + + +def _stacked_post_transfer_calibration_specs( + target_families: TargetFamilies, + *, + stage: str, +) -> tuple[PostTransferCalibrationSpec, ...]: + """Return the exact declared calibration specs present on one surface.""" + + if stage not in {"early_gap_fill", "late_transfer"}: + raise ValueError(f"Unknown stacked post-transfer calibration stage {stage!r}.") + surface_targets = _surface_target_triples(target_families) + return tuple( + spec + for spec in sorted( + post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values(), + key=lambda item: item.key, + ) + if spec.stage == stage + and any( + _stacked_calibration_family_matches( + spec, + entity=entity, + family=family, + target=target, + stage=stage, + ) + for entity, family, target in surface_targets + ) + ) + + +def _apply_stacked_post_transfer_calibrations( + before: Frame, + after: Frame, + *, + target_families: TargetFamilies, + stage: str, +) -> tuple[Frame, dict[str, dict[str, object]]]: + """Apply declared calibrations with owner-resolved origin and clone masks.""" + + current = after + receipts: dict[str, dict[str, object]] = {} + for spec in _stacked_post_transfer_calibration_specs( + target_families, + stage=stage, + ): + before_table = before.table(spec.entity) + table = current.table(spec.entity) + if not table.index.equals(before_table.index): + raise ValueError( + f"Post-transfer calibration {spec.key} requires byte-aligned " + "before/after entity indexes." + ) + channel = table[support_channel_column(spec.entity)].astype(str) + clone_index = pd.to_numeric( + table[support_clone_index_column(spec.entity)], + errors="raise", + ) + reference_rows = channel.eq(BASE_ASEC_SUPPORT_CHANNEL) & clone_index.eq(0) + recipient_rows = channel.eq(ACS_STACKED_SUPPORT_CHANNEL) & clone_index.eq(0) + mutable_rows = ( + recipient_rows + & before_table[spec.target].isna() + & table[spec.target].notna() + ) + allowed_rows: pd.Series | None = None + addition_rows: pd.Series | None = None + constraint_receipt: dict[str, object] = { + "constraint": spec.special_constraint, + } + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + qualifying = acs_transfer_runtime.acs_adult_care_qualifying_rows( + current.table("person") + ) + allowed_rows = mutable_rows & qualifying + addition_rows = _one_candidate_per_adult_care_tax_unit( + current, + mutable_rows=mutable_rows, + allowed_rows=allowed_rows, + ) + constraint_receipt.update( + { + "qualifying_mutable_rows": int(allowed_rows.sum()), + "one_per_empty_tax_unit_addition_candidates": int( + addition_rows.sum() + ), + } + ) + elif ( + spec.special_constraint + == "weeks_requires_positive_unemployment_compensation" + ): + person = current.table("person") + unemployment = pd.to_numeric( + person["unemployment_compensation"], + errors="raise", + ) + allowed_rows = mutable_rows & unemployment.gt(0.0) + addition_rows = allowed_rows.copy() + constraint_receipt["positive_unemployment_mutable_rows"] = int( + allowed_rows.sum() + ) + + reference_mask = reference_rows.to_numpy(dtype=bool) + recipient_mask = recipient_rows.to_numpy(dtype=bool) + mutable_mask = mutable_rows.to_numpy(dtype=bool) + allowed_mask = ( + mutable_mask if allowed_rows is None else allowed_rows.to_numpy(dtype=bool) + ) + addition_mask = ( + allowed_mask + if addition_rows is None + else addition_rows.to_numpy(dtype=bool) + ) + calibration_input = current + application = apply_post_transfer_calibration( + current, + entity=spec.entity, + family=spec.family, + target=spec.target, + reference_rows=reference_mask, + recipient_rows=recipient_mask, + mutable_rows=mutable_mask, + allowed_carrier_rows=(None if allowed_rows is None else allowed_mask), + addition_candidate_rows=(None if addition_rows is None else addition_mask), + ) + context_binding = _post_transfer_calibration_context_binding( + calibration_input, + application.frame, + entity=spec.entity, + target=spec.target, + reference_rows=reference_mask, + recipient_rows=recipient_mask, + mutable_rows=mutable_mask, + allowed_carrier_rows=allowed_mask, + addition_candidate_rows=addition_mask, + ) + validate_post_transfer_calibration_receipt( + application.receipt, + spec=spec, + boundary=f"stacked post-transfer owner {spec.key}", + expected_scope=context_binding["scope"], + expected_weights_sha256=str(context_binding["weights_sha256"]), + ) + current = application.frame + target_receipt: dict[str, object] = { + "stage": stage, + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": int(reference_rows.sum()), + "recipient_rows": int(recipient_rows.sum()), + "mutable_rows": int(mutable_rows.sum()), + "constraint": constraint_receipt, + "context_binding": context_binding, + "calibration": application.receipt, + } + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + person = current.table("person") + reconciled, reconciliation = acs_transfer_runtime.reconcile_acs_adult_care( + person, + mutable_rows=mutable_rows.to_numpy(dtype=bool), + ) + before_payload = _canonical_donor_series_payload( + person[spec.target], + boundary="adult-care post-calibration structure before verification", + ) + after_payload = _canonical_donor_series_payload( + reconciled, + boundary="adult-care post-calibration structure after verification", + ) + if before_payload != after_payload: + raise ValueError( + "Adult-care post-transfer calibration failed final " + "one-qualifying-carrier-per-tax-unit reconciliation." + ) + target_receipt["post_reconciliation"] = { + **reconciliation, + "status": "verified_no_op", + } + receipts[spec.key] = target_receipt + return current, receipts + + +def _attach_post_transfer_calibration_receipts( + target_receipts: dict[str, dict[str, object]], + calibrations: Mapping[str, Mapping[str, object]], +) -> dict[str, dict[str, object]]: + for key, calibration in calibrations.items(): + receipt_key = key + if receipt_key not in target_receipts: + entity, _family, target = key.split("/", 2) + candidates = [ + candidate + for candidate in target_receipts + if candidate.split("/", 2)[0] == entity + and candidate.split("/", 2)[2] == target + ] + if len(candidates) == 1: + receipt_key = candidates[0] + if receipt_key not in target_receipts: + raise ValueError( + f"Post-transfer calibration receipt target {key!r} is absent " + "from the transfer outcome receipt." + ) + target_receipts[receipt_key]["post_transfer_calibration"] = _json_ready( + calibration + ) + return target_receipts + + def gap_fill_stacked_spine( frame: Frame, *, @@ -7544,6 +9125,13 @@ def gap_fill_stacked_spine( ) -> GapFillResult: """Run the canonical stacked gap-fill plan with no caller authority.""" + if max_targets_per_fit != DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT: + raise ValueError( + "Canonical stacked gap fill requires " + f"max_targets_per_fit={DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT}; " + f"got {max_targets_per_fit}." + ) + return _gap_fill_stacked_spine_evaluate( frame, authority=_production_stacked_authority(), @@ -7656,6 +9244,10 @@ def _gap_fill_stacked_spine_evaluate( receipts: dict[str, object] = {} transfer_results: dict[str, AcsTransferResult] = {} for direction in directions: + direction_calibration_specs = _stacked_post_transfer_calibration_specs( + direction.target_families, + stage="early_gap_fill", + ) pre_counts = _verify_gap_fill_activation_authority( current, direction=direction, @@ -7679,21 +9271,51 @@ def _gap_fill_stacked_spine_evaluate( n_estimators=n_estimators, max_targets_per_fit=max_targets_per_fit, target_bank=(target_banks or {}).get(direction.name), + regime_evidence_targets=tuple( + (spec.entity, spec.target) for spec in direction_calibration_specs + ), ) + calibration_receipts: dict[str, dict[str, object]] = {} + if production: + calibrated, calibration_receipts = ( + _apply_stacked_post_transfer_calibrations( + current, + result.frame, + target_families=direction.target_families, + stage="early_gap_fill", + ) + ) + result = replace(result, frame=calibrated) transfer_results[direction.name] = result current = result.frame - receipts[direction.name] = _verify_gap_fill_outcome( + direction_receipt = _verify_gap_fill_outcome( current, direction=direction, pre_counts=pre_counts, donor_snapshot=donor_snapshot, result=result, ) + targets = direction_receipt["targets"] + assert isinstance(targets, dict) + _attach_post_transfer_calibration_receipts(targets, calibration_receipts) + direction_receipt["post_transfer_calibration"] = { + "policy_sha256": authority.post_transfer_calibration["sha256"], + "target_count": len(calibration_receipts), + "targets": sorted(calibration_receipts), + } + receipts[direction.name] = direction_receipt validate_stacked_spine_frame(current, boundary="stacked gap-fill output") + gap_fill_receipt = {"authority": authority_receipt, "directions": receipts} + if production: + validate_stacked_gap_fill_receipt( + gap_fill_receipt, + boundary="stacked gap-fill output", + frame=current, + ) return GapFillResult( frame=current, - receipt={"authority": authority_receipt, "directions": receipts}, + receipt=gap_fill_receipt, transfer_results=transfer_results, ) @@ -8078,6 +9700,13 @@ def _verify_gap_fill_outcome( } target_receipts: dict[str, dict[str, object]] = {} absence_rules = _direction_absence_rule_index(direction) + qrf_evidence_targets = { + (spec.entity, spec.target) + for spec in _stacked_post_transfer_calibration_specs( + direction.target_families, + stage="early_gap_fill", + ) + } for entity, families in direction.target_families.items(): table = frame.table(entity) channel = table[support_channel_column(entity)].astype(str) @@ -8093,6 +9722,7 @@ def _verify_gap_fill_outcome( }[entity] for family, targets in families.items(): for target in targets: + target_key = f"{entity}/{family}/{target}" label = f"{direction.name}/{entity}/{family}/{target}" before = donor_snapshot[entity].get(target) after = donor_after.get(target) @@ -8146,6 +9776,10 @@ def _verify_gap_fill_outcome( "unmodeled_rows": unmodeled, "residual_null_rows": residual_nulls, } + if (entity, target) in qrf_evidence_targets and record is not None: + target_receipt["qrf_pattern_evidence"] = ( + _acs_imputed_pattern_evidence(record) + ) rule = absence_rules.get((entity, target)) if rule is not None: expected_absence, absence_receipt = _gap_fill_absence_rule_mask( @@ -8173,7 +9807,7 @@ def _verify_gap_fill_outcome( f"residual_null_rows={residual_nulls}. Every downstream " "consumer requires this early target complete." ) - target_receipts[f"{entity}/{family}/{target}"] = target_receipt + target_receipts[target_key] = target_receipt if failures: raise ValueError( "Stacked gap-fill outcome verification failed:\n " + "\n ".join(failures) @@ -8429,7 +10063,23 @@ def _transfer_stacked_post_puf_inputs_evaluate( target_bank=target_bank, derive_schedule_d=derive_schedule_d, execution_contract=execution_contract, + regime_evidence_targets=tuple( + (spec.entity, spec.target) + for spec in _stacked_post_transfer_calibration_specs( + surface, + stage="late_transfer", + ) + ), ) + calibration_receipts: dict[str, dict[str, object]] = {} + if production: + calibrated, calibration_receipts = _apply_stacked_post_transfer_calibrations( + frame, + transfer.frame, + target_families=surface, + stage="late_transfer", + ) + transfer = replace(transfer, frame=calibrated) target_receipts = _verify_post_puf_transfer_outcome( transfer.frame, target_families=surface, @@ -8439,6 +10089,10 @@ def _transfer_stacked_post_puf_inputs_evaluate( producer_snapshot=producer_snapshot, result=transfer, ) + _attach_post_transfer_calibration_receipts( + target_receipts, + calibration_receipts, + ) validate_stacked_spine_frame( transfer.frame, boundary="stacked post-PUF transfer output", @@ -8455,6 +10109,11 @@ def _transfer_stacked_post_puf_inputs_evaluate( ), "resolved_donor_channel": transfer.resolved_donor_channel, "targets": target_receipts, + "post_transfer_calibration": { + "policy_sha256": authority.post_transfer_calibration["sha256"], + "target_count": len(calibration_receipts), + "targets": sorted(calibration_receipts), + }, "fit_records": [ {"fit_name": record.fit_name, "weight_kind": record.weight_kind} for record in transfer.fit_records @@ -8647,10 +10306,18 @@ def _verify_post_puf_transfer_outcome( target_receipts: dict[str, dict[str, object]] = {} puf_producer_keys = set(_surface_target_keys(puf_producer_families)) source_producer_keys = set(_surface_target_keys(source_producer_families)) + qrf_evidence_targets = { + (spec.entity, spec.target) + for spec in _stacked_post_transfer_calibration_specs( + target_families, + stage="late_transfer", + ) + } for entity, families in target_families.items(): table = frame.table(entity) for family, family_targets in families.items(): for target in family_targets: + target_receipt_key = f"{entity}/{family}/{target}" label = f"post_puf_transfer/{entity}/{family}/{target}" key = (entity, family, target, 0) puf_produced = key in puf_producer_keys @@ -8706,7 +10373,7 @@ def _verify_post_puf_transfer_outcome( f"unmodeled_rows={unmodeled}, " f"residual_null_rows={residual_nulls}; zero are allowed." ) - target_receipts[f"{entity}/{family}/{target}"] = { + target_receipt: dict[str, object] = { "producer_roles": [ role for role, active in ( @@ -8721,6 +10388,11 @@ def _verify_post_puf_transfer_outcome( "unmodeled_rows": unmodeled, "residual_null_rows": residual_nulls, } + if (entity, target) in qrf_evidence_targets and record is not None: + target_receipt["qrf_pattern_evidence"] = ( + _acs_imputed_pattern_evidence(record) + ) + target_receipts[target_receipt_key] = target_receipt if failures: raise ValueError( "Stacked post-PUF transfer outcome verification failed:\n " @@ -9170,6 +10842,7 @@ def _aggregate_late_transfer_result( validate_stacked_post_puf_transfer_receipt( receipt, boundary="US late-transfer DAG finalization", + frame=frame, ) return StackedPostPufTransferResult(frame, receipt, aggregate) diff --git a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py index 0fc2af81..e814cb47 100644 --- a/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py +++ b/packages/microcosm-build/src/microcosm/build/us_runtime/voluntary_filing.py @@ -385,7 +385,7 @@ def load_sipp_2023_voluntary_filing_donor( delimiter="|", usecols=list(SIPP_VOLUNTARY_FILING_SOURCE_COLUMNS), chunksize=int(chunksize), - low_memory=False, + low_memory=True, ) for chunk in reader: month = _numeric(chunk["MONTHCODE"]) diff --git a/packages/microcosm-build/tests/test_imputation_lineage_spec.py b/packages/microcosm-build/tests/test_imputation_lineage_spec.py index 93c699d6..9e7fb0fd 100644 --- a/packages/microcosm-build/tests/test_imputation_lineage_spec.py +++ b/packages/microcosm-build/tests/test_imputation_lineage_spec.py @@ -13,6 +13,9 @@ ACS_PERSON_TRANSFER_PREDICTORS, DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT, ) +from microcosm.build.us_runtime.post_transfer_calibration import ( + post_transfer_calibration_policy_identity, +) from microcosm.build.us_runtime.puf_support import PUF_TAX_DETAIL_DEFAULT_PREDICTORS from microcosm.build.us_runtime.stacked_spine import stacked_gap_fill_plan from microcosm.build.us_runtime.us_late_producer_registry import ( @@ -93,6 +96,13 @@ def test_model_and_split_attributes_match_constants_era_oracle( ), family["id"] +def test_post_draw_calibration_policy_matches_the_code( + spec: dict[str, object], +) -> None: + declared = spec["models"]["regime_gated_qrf"]["post_draw_calibration"] + assert declared == post_transfer_calibration_policy_identity() + + def test_non_transfer_producers_match_constants_era_registry( spec: dict[str, object], ) -> None: diff --git a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py index 59f0531b..dd92bc69 100644 --- a/packages/microcosm-build/tests/test_spec_engine_country_bundles.py +++ b/packages/microcosm-build/tests/test_spec_engine_country_bundles.py @@ -32,7 +32,7 @@ [ ( "be", - "262091db8c7b01b2a3b596aa2468d95855a63703ba9f8ebba2940cf5834c2c83", + "bf0221184046428782e7628dfad9b1a420bcc90c76ee88f4df373abecabff9d9", { "household.household_id", "person.person_id", @@ -42,7 +42,7 @@ ), ( "uk", - "1f163cbf7b35f07d49b6e2905d01a2519c0ba37e6d2d1470424272e18ac621db", + "8bf62b6e47583da1bdad1b71be1e705f424e6e245880e90f4411aba57fa5eb93", { "benunit.benunit_id", "household.household_id", diff --git a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py index c49b9a38..25f05295 100644 --- a/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py +++ b/packages/microcosm-build/tests/test_spec_engine_coverage_tool.py @@ -52,21 +52,21 @@ def test_us_coverage_is_exact_complete_and_honest( assert_coverage_complete(coverage_report) assert coverage_report["status"] == "pass" fields = coverage_report["field_usage"] - assert fields["configuration_field_count"] == 41_379 - assert fields["authored_normative_field_count"] == 32_160 + assert fields["configuration_field_count"] == 41_471 + assert fields["authored_normative_field_count"] == 32_252 assert fields["resolved_binding_field_count"] == 9_219 - assert fields["consumed_field_count"] == 41_379 + assert fields["consumed_field_count"] == 41_471 assert fields["unused_field_count"] == 0 assert fields["multiple_primary_use_field_count"] == 0 assert fields["claim_count"] == 47 assert fields["mode_counts"] == { - "legacy_behavior": 13_244, + "legacy_behavior": 13_336, "compiler_semantic": 27_661, "front_end_validation": 346, "identity_only": 128, } assert fields["generation0_effect_counts"] == { - "legacy_behavior": 37_717, + "legacy_behavior": 37_809, "no_generation0_effect": 3_662, } diff --git a/packages/microcosm-build/tests/test_spec_engine_field_usage.py b/packages/microcosm-build/tests/test_spec_engine_field_usage.py index e0f565e0..73721377 100644 --- a/packages/microcosm-build/tests/test_spec_engine_field_usage.py +++ b/packages/microcosm-build/tests/test_spec_engine_field_usage.py @@ -92,22 +92,22 @@ def _mutated_bundle( def test_exact_complete_ledger_has_one_primary_mode_per_pointer(field_ledger) -> None: - assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 41_379 + assert len(field_ledger.fields) == EXPECTED_CONFIGURATION_FIELD_COUNT == 41_471 assert field_ledger.source_counts == { - "authored": 32_160, + "authored": 32_252, "resolved_bindings": 9_219, } assert field_ledger.mode_counts == { - "legacy_behavior": 13_244, + "legacy_behavior": 13_336, "compiler_semantic": 27_661, "front_end_validation": 346, "identity_only": 128, } assert field_ledger.generation0_effect_counts == { - "legacy_behavior": 37_717, + "legacy_behavior": 37_809, "no_generation0_effect": 3_662, } - assert len({field.pointer for field in field_ledger.fields}) == 41_379 + assert len({field.pointer for field in field_ledger.fields}) == 41_471 def test_eligibility_concepts_are_validation_not_generation0_behavior( diff --git a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py index 1a514700..2545563f 100644 --- a/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py +++ b/packages/microcosm-build/tests/test_spec_engine_inventory_coverage.py @@ -65,7 +65,7 @@ EXPECTED_COUNTS = { "adapter_surfaces": 13, - "authority_components": 8, + "authority_components": 9, "early_families": 13, "early_targets": 48, "itemization_batches": 5, diff --git a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py index 86d65958..9cb7f29c 100644 --- a/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py +++ b/packages/microcosm-build/tests/test_spec_engine_legacy_adapter.py @@ -151,9 +151,7 @@ def test_compiled_gate_is_byte_identical_to_constants_era_payloads( "gap_fill_producer_schedule_receipt": imputation[ "gap_fill_producer_schedule_receipt" ], - "late_producer_schedule_receipt": imputation[ - "late_producer_schedule_receipt" - ], + "late_producer_schedule_receipt": imputation["late_producer_schedule_receipt"], "overlap_ownership": imputation["overlap_ownership"], } live_gate = { @@ -197,7 +195,7 @@ def test_adapter_preserves_generation_zero_identity_components( imputation = legacy_payload["imputation"] assert isinstance(imputation, dict) assert legacy_payload["stacked_authority_receipt"]["sha256"] == ( - "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "24ae81f408e8d608db1956463206fe9842b5214ca813113d3e5366fc66ca3cde" ) assert imputation["late_producer_schedule_receipt"]["schedule_sha256"] == ( "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5" diff --git a/packages/microcosm-build/tests/test_spec_engine_loader.py b/packages/microcosm-build/tests/test_spec_engine_loader.py index e6a5391a..eda92b0d 100644 --- a/packages/microcosm-build/tests/test_spec_engine_loader.py +++ b/packages/microcosm-build/tests/test_spec_engine_loader.py @@ -235,7 +235,7 @@ def test_semantic_hash_has_golden_vector_and_surface_separation(tmp_path) -> Non # Pin the domain separator, normalization rules, schema-set receipt, and # exact normative projection as one reviewable golden vector. assert first.spec_sha256 == ( - "9afebedeebc4194243e8307703ebc0feac975a69a186d28e266f1c4ef356bbe3" + "b1ab6ab000689cc03e1088d422d7b6328b9bff2878f39448a529e5513c03ed14" ) second_root = _rich_minimal(tmp_path / "xy", note="second", store="local:b") diff --git a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py index 5c236bc7..ff332872 100644 --- a/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py +++ b/packages/microcosm-build/tests/test_spec_engine_stacked_authority_semantics.py @@ -30,8 +30,7 @@ ROOT = Path(__file__).resolve().parents[3] MODULE = ( - ROOT - / "packages/microcosm-build/src/microcosm/build/spec_engine/" + ROOT / "packages/microcosm-build/src/microcosm/build/spec_engine/" "stacked_authority_semantics.py" ) @@ -74,11 +73,10 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer assert projected == live assert stacked_identity_bytes(projected) == _canonical_bytes(live) assert projected["sha256"] == ( - "f0b676f6508dbf6bb2b787c42e6b85331bacc57c6649ac7ad15fdaa5884a1b2d" + "24ae81f408e8d608db1956463206fe9842b5214ca813113d3e5366fc66ca3cde" ) assert { - name: component["sha256"] - for name, component in projected["components"].items() + name: component["sha256"] for name, component in projected["components"].items() } == { "declared_surface": ( "d6d987b34d88bf4a6327dcd4318c67d45ab9392a1bdb2f586887fdcf340faffc" @@ -98,6 +96,9 @@ def test_authority_projection_is_field_and_byte_identical_to_live_generation_zer "post_puf_transfer_surface": ( "290c871baf8d5602dc1009fbba3db0b7fcf65b60ccf6557bf7776b8c0c07dbec" ), + "post_transfer_calibration": ( + "141519684c72ab84a077ae0f5716a0416f1e19da57262948e459633cbe560576" + ), "puf_capital_gains_tail_support_contract": ( "91bc9272cb7f28c6271fb9695ddf6ec05fe55b4458070fc47ae4a1d9607f3c89" ), @@ -169,9 +170,10 @@ def test_checkpoint_projection_is_field_and_byte_identical_to_live_oracle( assert projected == live assert stacked_identity_bytes(projected) == _canonical_bytes(live) assert list(projected["inputs"]) == ["alpha", "zeta"] - assert projected["pool_code"]["late_producer_schedule"][ - "schedule_sha256" - ] == "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5" + assert ( + projected["pool_code"]["late_producer_schedule"]["schedule_sha256"] + == "b1d00afea69b2009d862ca73fff1b63ce56628a8a0790be49918e4bbbecc9fc5" + ) def test_static_projection_selects_exact_defaulted_live_identity_components( diff --git a/packages/microcosm-build/tests/test_us_acs_multispine.py b/packages/microcosm-build/tests/test_us_acs_multispine.py index 3d166a00..98b21c6a 100644 --- a/packages/microcosm-build/tests/test_us_acs_multispine.py +++ b/packages/microcosm-build/tests/test_us_acs_multispine.py @@ -1,6 +1,7 @@ from __future__ import annotations import json +from dataclasses import replace from pathlib import Path from types import SimpleNamespace from typing import Any, cast @@ -12,7 +13,10 @@ from microcosm.build.gates import FitWeightRecord from microcosm.build.us_runtime import acs_multispine from microcosm.build.us_runtime.acs_pums import AcsPumsSource -from microcosm.build.us_runtime.acs_transfer import AcsImputedInput +from microcosm.build.us_runtime.acs_transfer import ( + AcsImputedInput, + AcsTransferPattern, +) from microcosm.build.us_runtime.base_pool import spine_column from microcosm.build.us_runtime.puma_ladder import UsPumaLadder from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights @@ -57,6 +61,31 @@ def test__given_no_source__then_base_frame_is_an_untouched_identity( json.dumps(result.provenance, allow_nan=False) +def test_json_ready_omits_empty_opt_in_transfer_pattern_regimes() -> None: + pattern = AcsTransferPattern( + name="fixture", + observed_optional_predictors=(), + predictors=("age",), + seed=1, + weight_kind="source", + donor_rows=2, + recipient_rows=1, + ) + + legacy = acs_multispine._json_ready(pattern) + selected = acs_multispine._json_ready( + replace( + pattern, + target_regimes=(("fixture_target", "positive_only"),), + ) + ) + + assert isinstance(legacy, dict) + assert "target_regimes" not in legacy + assert isinstance(selected, dict) + assert selected["target_regimes"] == [["fixture_target", "positive_only"]] + + def test__given_source__then_stages_run_in_order_and_provenance_is_json_ready( monkeypatch, tmp_path, diff --git a/packages/microcosm-build/tests/test_us_acs_transfer.py b/packages/microcosm-build/tests/test_us_acs_transfer.py index ff4c7fc6..1744919a 100644 --- a/packages/microcosm-build/tests/test_us_acs_transfer.py +++ b/packages/microcosm-build/tests/test_us_acs_transfer.py @@ -21,6 +21,7 @@ ACS_OPTIONAL_PERSON_TRANSFER_PREDICTORS, ACS_PERSON_TRANSFER_PREDICTORS, AcsTransferResult, + acs_adult_care_qualifying_rows, declared_acs_transfer_target_families, default_acs_transfer_target_families, transfer_acs_inputs, @@ -30,6 +31,7 @@ ) from microcosm.build.us_runtime.puf_support import clone_us_frame_for_puf_support from microcosm.build.us_runtime.spine_assembly import assemble_spines +from microcosm.fit import Regime from microcosm.frame import US_SCHEMA, EntitySchema, Frame, WeightKind, Weights from microcosm.frame.adapters.policyengine_us import ( PolicyEngineUSVariableMetadataIndex, @@ -497,6 +499,7 @@ def _run_bank_fixture( seed=37, n_estimators=1, target_bank=target_bank, + regime_evidence_targets=tuple(("person", target) for target in _BANK_TARGETS), ) @@ -902,6 +905,151 @@ def test_large_target_family_is_split_to_bound_retained_qrf_forests( "wide_numeric__batch_3", "wide_numeric__batch_4", } + assert all( + not pattern.target_regimes + for entry in result.imputed_inputs + for pattern in entry.patterns + ) + + +def test_pattern_provenance_records_ordered_exact_donor_target_regimes( + monkeypatch: pytest.MonkeyPatch, +) -> None: + targets = ( + "fixture_positive_only", + "fixture_zero_inflated_positive", + "fixture_three_sign", + ) + selected_targets = (targets[0], targets[2]) + donor = _with_columns( + _donor_frame(), + "person", + { + targets[0]: np.arange(1.0, 9.0), + targets[1]: [0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0], + targets[2]: [-3.0, -2.0, -1.0, 0.0, 0.0, 1.0, 2.0, 3.0], + }, + ) + recipient = _recipient_frame() + for column in ( + "employment_income_before_lsr", + "self_employment_income_before_lsr", + ): + donor = _drop_column(donor, "person", column) + recipient = _drop_column(recipient, "person", column) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + donor, + target_families={"person": {"fixture_regimes": targets}}, + n_estimators=1, + regime_evidence_targets=tuple( + ("person", target) for target in selected_targets + ), + ) + + expected = ( + (targets[0], Regime.POSITIVE_ONLY), + (targets[2], Regime.THREE_SIGN), + ) + assert len(result.imputed_inputs[0].patterns) == 1 + assert { + item.column: item.patterns[0].target_regimes for item in result.imputed_inputs + } == { + targets[0]: expected, + targets[1]: (), + targets[2]: expected, + } + + +def test_selected_regime_fit_keeps_complete_selected_sibling( + monkeypatch: pytest.MonkeyPatch, +) -> None: + targets = ("fixture_selected_complete", "fixture_selected_missing") + donor = _with_columns( + _donor_frame(), + "person", + { + targets[0]: np.arange(1.0, 9.0), + targets[1]: [0.0, 0.0, 1.0, 1.0, 2.0, 2.0, 3.0, 3.0], + }, + ) + recipient = _with_columns( + _recipient_frame(), + "person", + {targets[0]: np.full(6, 99.0)}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + _MeanQRF.calls = [] + + result = transfer_acs_inputs( + recipient, + donor, + target_families={"person": {"selected_siblings": targets}}, + n_estimators=1, + regime_evidence_targets=tuple(("person", target) for target in targets), + ) + + assert [record.column for record in result.imputed_inputs] == [targets[1]] + assert all( + tuple(target for target, _regime in pattern.target_regimes) == targets + for pattern in result.imputed_inputs[0].patterns + ) + assert result.frame.table("person")[targets[0]].eq(99.0).all() + + +def test_selected_regime_fit_does_not_expand_mixed_family_or_change_unselected_draw( + monkeypatch: pytest.MonkeyPatch, +) -> None: + targets = ( + "fixture_selected_complete", + "fixture_selected_missing", + "fixture_unselected_missing", + ) + donor = _with_columns( + _donor_frame(), + "person", + { + target: np.arange(1.0, 9.0) + position + for position, target in enumerate(targets) + }, + ) + recipient = _with_columns( + _recipient_frame(), + "person", + {targets[0]: np.full(6, 99.0)}, + ) + monkeypatch.setattr(acs_transfer_module, "QRF", _MeanQRF) + + baseline = transfer_acs_inputs( + recipient, + donor, + target_families={"person": {"mixed_selection": targets}}, + n_estimators=1, + ) + selected = transfer_acs_inputs( + recipient, + donor, + target_families={"person": {"mixed_selection": targets}}, + n_estimators=1, + regime_evidence_targets=( + ("person", targets[0]), + ("person", targets[1]), + ), + ) + + pd.testing.assert_series_equal( + selected.frame.table("person")[targets[2]], + baseline.frame.table("person")[targets[2]], + ) + records = {record.column: record for record in selected.imputed_inputs} + assert all(not pattern.target_regimes for pattern in records[targets[2]].patterns) + assert all( + tuple(target for target, _regime in pattern.target_regimes) == (targets[1],) + for pattern in records[targets[1]].patterns + ) def test_target_bank_cold_output_matches_unbanked_monolith( @@ -915,6 +1063,16 @@ def test_target_bank_cold_output_matches_unbanked_monolith( banked = _run_bank_fixture(bank) _assert_transfer_results_exact(banked, monolithic) + ordinary_patterns = monolithic.imputed_inputs[0].patterns + banked_patterns = banked.imputed_inputs[0].patterns + assert ordinary_patterns + assert tuple(pattern.target_regimes for pattern in banked_patterns) == tuple( + pattern.target_regimes for pattern in ordinary_patterns + ) + assert all( + tuple(target for target, _regime in pattern.target_regimes) == _BANK_TARGETS + for pattern in ordinary_patterns + ) _assert_bank_receipt( bank, sources=("rebuilt", "rebuilt", "rebuilt"), @@ -1045,6 +1203,11 @@ def run( person[target].dtype == CANONICAL_STRING_DTYPE for target in ("ssn_card_type", "immigration_status_str") ) + assert all( + not pattern.target_regimes + for record in result.imputed_inputs + for pattern in record.patterns + ) targets = warm_bank.receipt()["targets"] assert set(targets) == {"0"} @@ -2381,3 +2544,54 @@ def test_adult_care_reconciliation_enforces_statute_structure() -> None: "cleared_multi_carrier_rows": 1, "remaining_carriers": 2, } + + +def test_adult_care_qualifying_rows_enforces_role_and_marriage_rules() -> None: + person = pd.DataFrame( + { + "is_incapable_of_self_care": [ + True, + False, + True, + True, + True, + True, + ], + "tax_unit_role_input": [ + "DEPENDENT", + "DEPENDENT", + "HEAD", + "SPOUSE", + "HEAD", + "OTHER", + ], + "person_tax_unit_id": [1, 1, 2, 2, 3, 3], + } + ) + + qualifying = acs_adult_care_qualifying_rows(person) + + assert qualifying.tolist() == [True, False, True, True, False, False] + + +@pytest.mark.parametrize( + "missing_column", + [ + "is_incapable_of_self_care", + "tax_unit_role_input", + "person_tax_unit_id", + ], +) +def test_adult_care_qualifying_rows_fails_closed_on_missing_structure( + missing_column: str, +) -> None: + person = pd.DataFrame( + { + "is_incapable_of_self_care": [True], + "tax_unit_role_input": ["DEPENDENT"], + "person_tax_unit_id": [1], + } + ).drop(columns=missing_column) + + with pytest.raises(ValueError, match=missing_column): + acs_adult_care_qualifying_rows(person) diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py index 8f5dc55e..28c27e1f 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_h5_io.py @@ -9,6 +9,7 @@ import pytest import microcosm.build.us_runtime.h5_io as h5_io +import microcosm.build.us_runtime.post_transfer_calibration as post_transfer_calibration_runtime import microcosm.build.us_runtime.stacked_spine as stacked_spine_module from microcosm.build.frame_checkpoint import ( load_frame_checkpoint, @@ -507,6 +508,100 @@ def _write_ready_pool(tmp_path: Path, *, stacked: bool = False) -> Path: return manifest_path +def _canonical_late_calibration_owner_receipt( + spec: post_transfer_calibration_runtime.PostTransferCalibrationSpec, +) -> dict[str, object]: + values = np.asarray( + [10.0, 20.0, 30.0, 40.0, 50.0, 100.0, 200.0, 300.0, 400.0, 500.0] + ) + weights = np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0]) + entity_ids = np.arange(1, len(values) + 1) + reference = np.asarray([True] * 5 + [False] * 5) + recipient = ~reference + constrained = spec.special_constraint != "none" + result = post_transfer_calibration_runtime.calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient if constrained else None, + addition_candidate_rows=recipient if constrained else None, + ) + calibration = result.receipt + scope = calibration["scope"] + constraint: dict[str, object] = {"constraint": spec.special_constraint} + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + constraint.update( + { + "qualifying_mutable_rows": scope["allowed_carrier_rows"], + "one_per_empty_tax_unit_addition_candidates": scope[ + "addition_candidate_rows" + ], + } + ) + elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": + constraint["positive_unemployment_mutable_rows"] = scope["allowed_carrier_rows"] + owner: dict[str, object] = { + "stage": "late_transfer", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": scope["reference_rows"], + "recipient_rows": scope["recipient_rows"], + "mutable_rows": scope["mutable_rows"], + "constraint": constraint, + "context_binding": { + "scope": dict(scope), + "weights_sha256": calibration["weights"]["sha256"], + "live_output": { + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "reference_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + entity_ids[reference] + ) + ), + "recipient_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + entity_ids[recipient] + ) + ), + "reference_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + result.values[reference], + boundary="synthetic reference calibration output", + ) + ), + "recipient_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + result.values[recipient], + boundary="synthetic recipient calibration output", + ) + ), + "reference_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[reference], + boundary="synthetic reference calibration weights", + ) + ), + "recipient_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[recipient], + boundary="synthetic recipient calibration weights", + ) + ), + }, + }, + "calibration": calibration, + } + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + owner["post_reconciliation"] = {"status": "verified_no_op"} + return owner + + def _canonical_stacked_late_dag_receipt() -> dict[str, object]: """Build a signed fixture receipt over the live canonical contracts.""" @@ -547,21 +642,44 @@ def _canonical_stacked_late_dag_receipt() -> dict[str, object]: } }, } - group_receipts = { - group.name: { + late_specs = { + spec.key: spec + for spec in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "late_transfer" + } + policy_sha256 = ( + post_transfer_calibration_runtime.post_transfer_calibration_policy_identity()[ + "sha256" + ] + ) + group_receipts: dict[str, object] = {} + for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS: + group_targets = { + f"{group.entity}/{group.family}/{target}": { + "authorized_null_rows": 0, + "imputed_rows": 0, + "unmodeled_rows": 0, + "residual_null_rows": 0, + } + for target in group.targets + } + calibrated_keys = sorted(set(group_targets) & set(late_specs)) + for key in calibrated_keys: + group_targets[key]["post_transfer_calibration"] = ( + _canonical_late_calibration_owner_receipt(late_specs[key]) + ) + group_receipts[group.name] = { "producer": group.name, "entity": group.entity, "family": group.family, "ordered_targets": list(group.targets), - "targets": { - f"{group.entity}/{group.family}/{target}": { - "residual_null_rows": 0, - } - for target in group.targets + "targets": group_targets, + "post_transfer_calibration": { + "policy_sha256": policy_sha256, + "target_count": len(calibrated_keys), + "targets": calibrated_keys, }, } - for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS - } group_by_name = { group.name: group for group in stacked_spine_module.CANONICAL_US_LATE_TRANSFER_GROUPS diff --git a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py index 1031564d..6ce817b2 100644 --- a/packages/microcosm-build/tests/test_us_multispine_pool_tool.py +++ b/packages/microcosm-build/tests/test_us_multispine_pool_tool.py @@ -23,6 +23,7 @@ import microcosm.build.us_runtime.acs_transfer as acs_transfer_module import microcosm.build.us_runtime.multispine_pool as multispine_pool_module +import microcosm.build.us_runtime.post_transfer_calibration as post_transfer_calibration_runtime import microcosm.build.us_runtime.stacked_spine as stacked_spine_module from microcosm.build.gates import GateReport, GateResult from microcosm.build.logbook import LOGBOOK_ROW_FIELDS, load_logbook_row @@ -914,10 +915,187 @@ def _noncanonical_post_puf_authority_receipt() -> dict[str, object]: return stacked_spine_module._authority_receipt(test_authority) +def _canonical_late_calibration_owner_receipt( + spec: post_transfer_calibration_runtime.PostTransferCalibrationSpec, + *, + frame: Frame | None = None, +) -> dict[str, object]: + if frame is not None: + table = frame.table(spec.entity) + channel = table[support_channel_column(spec.entity)].astype(str) + clone_index = pd.to_numeric( + table[support_clone_index_column(spec.entity)], + errors="raise", + ) + reference = (channel.eq("asec") & clone_index.eq(0)).to_numpy(dtype=bool) + recipient = (channel.eq("acs") & clone_index.eq(0)).to_numpy(dtype=bool) + constrained = spec.special_constraint != "none" + application = post_transfer_calibration_runtime.apply_post_transfer_calibration( + frame, + entity=spec.entity, + family=spec.family, + target=spec.target, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient if constrained else None, + addition_candidate_rows=recipient if constrained else None, + ) + before_values = table[spec.target].to_numpy(copy=False) + after_values = application.frame.table(spec.entity)[spec.target].to_numpy( + copy=False + ) + if not np.array_equal( + before_values.view(np.uint64), + after_values.view(np.uint64), + ): + raise AssertionError( + f"Live calibration fixture unexpectedly changed {spec.key}." + ) + calibration = application.receipt + scope = calibration["scope"] + constraint: dict[str, object] = {"constraint": spec.special_constraint} + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + constraint.update( + { + "qualifying_mutable_rows": scope["allowed_carrier_rows"], + "one_per_empty_tax_unit_addition_candidates": scope[ + "addition_candidate_rows" + ], + } + ) + elif ( + spec.special_constraint + == "weeks_requires_positive_unemployment_compensation" + ): + constraint["positive_unemployment_mutable_rows"] = scope[ + "allowed_carrier_rows" + ] + owner: dict[str, object] = { + "stage": "late_transfer", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": scope["reference_rows"], + "recipient_rows": scope["recipient_rows"], + "mutable_rows": scope["mutable_rows"], + "constraint": constraint, + "context_binding": ( + stacked_spine_module._post_transfer_calibration_context_binding( + frame, + application.frame, + entity=spec.entity, + target=spec.target, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient, + addition_candidate_rows=recipient, + ) + ), + "calibration": calibration, + } + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + owner["post_reconciliation"] = {"status": "verified_no_op"} + return owner + + values = np.asarray( + [10.0, 20.0, 30.0, 40.0, 50.0, 100.0, 200.0, 300.0, 400.0, 500.0] + ) + weights = np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0]) + reference = np.asarray([True] * 5 + [False] * 5) + recipient = ~reference + constrained = spec.special_constraint != "none" + calibration_result = ( + post_transfer_calibration_runtime.calibrate_post_transfer_values( + values, + weights, + np.arange(1, len(values) + 1), + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient if constrained else None, + addition_candidate_rows=recipient if constrained else None, + ) + ) + calibration = calibration_result.receipt + scope = calibration["scope"] + constraint: dict[str, object] = {"constraint": spec.special_constraint} + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + constraint.update( + { + "qualifying_mutable_rows": scope["allowed_carrier_rows"], + "one_per_empty_tax_unit_addition_candidates": scope[ + "addition_candidate_rows" + ], + } + ) + elif spec.special_constraint == "weeks_requires_positive_unemployment_compensation": + constraint["positive_unemployment_mutable_rows"] = scope["allowed_carrier_rows"] + owner: dict[str, object] = { + "stage": "late_transfer", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": scope["reference_rows"], + "recipient_rows": scope["recipient_rows"], + "mutable_rows": scope["mutable_rows"], + "constraint": constraint, + "context_binding": { + "scope": dict(scope), + "weights_sha256": calibration["weights"]["sha256"], + "live_output": { + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "reference_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + np.arange(1, len(values) + 1)[reference] + ) + ), + "recipient_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + np.arange(1, len(values) + 1)[recipient] + ) + ), + "reference_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + calibration_result.values[reference], + boundary="synthetic reference calibration output", + ) + ), + "recipient_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + calibration_result.values[recipient], + boundary="synthetic recipient calibration output", + ) + ), + "reference_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[reference], + boundary="synthetic reference calibration weights", + ) + ), + "recipient_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[recipient], + boundary="synthetic recipient calibration weights", + ) + ), + }, + }, + "calibration": calibration, + } + if spec.special_constraint == "adult_care_qualifying_one_per_tax_unit": + owner["post_reconciliation"] = {"status": "verified_no_op"} + return owner + + def _canonical_late_transfer_receipt( pool_tool: ModuleType, *, authority: Mapping[str, object] | None = None, + frame: Frame | None = None, ) -> dict[str, object]: canonical_family = { (entity, target): family @@ -929,15 +1107,43 @@ def _canonical_late_transfer_receipt( } groups: dict[str, object] = {} targets: dict[str, object] = {} + late_specs = { + spec.key: spec + for spec in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "late_transfer" + } + policy_sha256 = ( + post_transfer_calibration_runtime.post_transfer_calibration_policy_identity()[ + "sha256" + ] + ) for group in pool_tool.CANONICAL_US_LATE_TRANSFER_GROUPS: group_targets = { - f"{group.entity}/{group.family}/{target}": {"residual_null_rows": 0} + f"{group.entity}/{group.family}/{target}": { + "authorized_null_rows": 0, + "imputed_rows": 0, + "unmodeled_rows": 0, + "residual_null_rows": 0, + } for target in group.targets } + calibrated_keys = sorted(set(group_targets) & set(late_specs)) + for key in calibrated_keys: + group_targets[key]["post_transfer_calibration"] = ( + _canonical_late_calibration_owner_receipt( + late_specs[key], + frame=frame, + ) + ) groups[group.name] = { "producer": group.name, "ordered_targets": list(group.targets), "targets": group_targets, + "post_transfer_calibration": { + "policy_sha256": policy_sha256, + "target_count": len(calibrated_keys), + "targets": calibrated_keys, + }, } for target in group.targets: targets[ @@ -974,6 +1180,7 @@ def _canonical_late_dag_receipt( *, authority: Mapping[str, object] | None = None, output_frame_sha256: str = "f" * 64, + frame: Frame | None = None, ) -> dict[str, object]: schedule = stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE schedule_receipt = pool_tool._json_ready( @@ -1016,6 +1223,7 @@ def _canonical_late_dag_receipt( transfer = _canonical_late_transfer_receipt( pool_tool, authority=authority, + frame=frame, ) input_frame_sha256 = "e" * 64 previous_sha256 = stacked_spine_module._late_execution_genesis_sha256( @@ -1239,10 +1447,50 @@ def _authorized_late_impute_fixture( ) -> tuple[Frame, dict[str, object], str]: """Bind one structurally signed synthetic DAG proof to a live fixture frame.""" + tables = {entity: frame.table(entity).copy(deep=True) for entity in frame.entities} + for entity, table in tables.items(): + if support_channel_column(entity) not in table: + table[support_channel_column(entity)] = np.resize( + np.asarray(["asec", "acs"], dtype=object), + len(table), + ) + if support_clone_index_column(entity) not in table: + table[support_clone_index_column(entity)] = np.zeros( + len(table), dtype=np.int64 + ) + for ( + spec + ) in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values(): + if spec.stage == "late_transfer" and spec.target not in tables[spec.entity]: + tables[spec.entity][spec.target] = np.ones( + len(tables[spec.entity]), dtype=np.float64 + ) + person = tables["person"] + person["unemployment_compensation"] = np.ones(len(person), dtype=np.float64) + person["is_incapable_of_self_care"] = pd.Series( + True, + index=person.index, + dtype="boolean", + ) + person["tax_unit_role_input"] = pd.Series( + "DEPENDENT", + index=person.index, + dtype="string", + ) + tables.update({name: frame.link(name) for name in frame.links}) + frame = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) dag = _canonical_late_dag_receipt( pool_tool, authority=authority, output_frame_sha256=stacked_spine_module._late_frame_content_sha256(frame), + frame=frame, ) authorized, transition_authority_sha256 = ( stacked_spine_module._bind_late_producer_transition_authority(frame, dag) @@ -1477,16 +1725,64 @@ def late_producer_dag(frame: Frame, **kwargs: object): "family": group.family, "ordered_targets": list(group.targets), } + late_tables = { + entity: primary_puf_result.frame.table(entity).copy(deep=True) + for entity in primary_puf_result.frame.entities + } + for ( + spec + ) in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values(): + if ( + spec.stage == "late_transfer" + and spec.target not in late_tables[spec.entity] + ): + late_tables[spec.entity][spec.target] = np.ones( + len(late_tables[spec.entity]), + dtype=np.float64, + ) + late_person = late_tables["person"] + late_person["unemployment_compensation"] = np.ones( + len(late_person), + dtype=np.float64, + ) + late_person["is_incapable_of_self_care"] = pd.Series( + True, + index=late_person.index, + dtype="boolean", + ) + late_person["tax_unit_role_input"] = pd.Series( + "DEPENDENT", + index=late_person.index, + dtype="string", + ) + late_tables.update( + { + name: primary_puf_result.frame.link(name) + for name in primary_puf_result.frame.links + } + ) + late_frame = Frame( + late_tables, + primary_puf_result.frame.schema, + { + entity: primary_puf_result.frame.weights_for(entity) + for entity in primary_puf_result.frame.weighted_entities + }, + primary_puf_result.frame.strata, + mass_log=primary_puf_result.frame.mass_log, + metadata=primary_puf_result.frame.metadata, + ) dag_receipt = _canonical_late_dag_receipt( pool_tool, authority=post_puf_authority, output_frame_sha256=stacked_spine_module._late_frame_content_sha256( - primary_puf_result.frame + late_frame ), + frame=late_frame, ) authorized_frame, transition_authority_sha256 = ( stacked_spine_module._bind_late_producer_transition_authority( - primary_puf_result.frame, + late_frame, dag_receipt, ) ) @@ -1719,10 +2015,20 @@ def test_stacked_tool_entrypoint_fixture_e2e_emits_one_logbook_row_at_every_term published_dag = manifest["stage_receipts"]["impute"][ "stacked_late_producer_dag" ] - assert published_dag == _canonical_late_dag_receipt( - pool_tool, - output_frame_sha256=published_dag["output_frame_sha256"], - ) + published_transfer = published_dag["post_puf_transfer"] + assert published_transfer["completion"] == { + "status": "complete", + "group_count": 19, + "target_count": 70, + "residual_null_rows": 0, + } + calibrated = [ + target["post_transfer_calibration"] + for target in published_transfer["targets"].values() + if "post_transfer_calibration" in target + ] + assert len(calibrated) == 7 + assert all(owner["context_binding"]["live_output"] for owner in calibrated) expected_late_authority_sha256 = ( stacked_spine_module._late_producer_transition_authority_receipt( published_dag @@ -1903,7 +2209,7 @@ def capture_equality(expected: object, actual: object) -> None: "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "586491f0866180f7a8f1e01530af5c2a3f2ebc18ad289ac1a3385d1897e79626", + "spec_sha256": "d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0", }, } @@ -2106,7 +2412,7 @@ def test_constants_adapter_fixture_checkpoints_are_byte_identical_and_only_recei "country": "us", "schema_id": "country_spec", "schema_version": 1, - "spec_sha256": "586491f0866180f7a8f1e01530af5c2a3f2ebc18ad289ac1a3385d1897e79626", + "spec_sha256": "d3de6760727cfcb6800209670d37e02b373d8dcda19f8ad054aa9d410e0efbb0", } def run_fixture(root: Path, *, config_authority: str) -> dict[str, object]: @@ -2357,6 +2663,127 @@ def test_late_dag_validator_rejects_forged_execution_row( ) +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ( + "stripped_group_summary", + "stripped or misbound calibration summary evidence", + ), + ("stripped_target_evidence", "calibration evidence is absent"), + ("malformed_target_evidence", "calibration receipt digest is invalid"), + ("deleted_target_receipt", "target surface is non-canonical"), + ("extra_target_receipt", "target surface is non-canonical"), + ("stripped_owner_count", "reference_rows evidence is misbound"), + ("stripped_live_output", "live-output binding is absent"), + ), +) +def test_late_transfer_validator_rejects_stripped_calibration_evidence( + pool_tool: ModuleType, + mutation: str, + error_match: str, +) -> None: + receipt = _canonical_late_transfer_receipt(pool_tool) + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="canonical calibration evidence control", + ) + forged = copy.deepcopy(receipt) + group_receipt = next( + group + for group in forged["groups"].values() + if group["post_transfer_calibration"]["target_count"] > 0 + ) + if mutation == "stripped_group_summary": + group_receipt.pop("post_transfer_calibration") + else: + target_key = group_receipt["post_transfer_calibration"]["targets"][0] + if mutation == "deleted_target_receipt": + group_receipt["targets"].pop(target_key) + elif mutation == "extra_target_receipt": + group_receipt["targets"]["person/forged/extra_target"] = {} + elif mutation == "stripped_owner_count": + group_receipt["targets"][target_key]["post_transfer_calibration"].pop( + "reference_rows" + ) + elif mutation == "stripped_live_output": + group_receipt["targets"][target_key]["post_transfer_calibration"][ + "context_binding" + ].pop("live_output") + elif mutation == "stripped_target_evidence": + target_receipt = group_receipt["targets"][target_key] + target_receipt.pop("post_transfer_calibration") + else: + target_receipt = group_receipt["targets"][target_key] + target_receipt["post_transfer_calibration"]["calibration"]["sha256"] = ( + "0" * 64 + ) + + with pytest.raises(ValueError, match=error_match): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + forged, + boundary=f"{mutation} regression", + ) + + +@pytest.mark.parametrize( + ("target", "constraint_column", "replacement", "error_match"), + ( + ( + "weeks_unemployed", + "unemployment_compensation", + 0.0, + "positive weeks-unemployed carriers lack positive unemployment", + ), + ( + "pre_subsidy_care_expenses", + "is_incapable_of_self_care", + False, + "live adult-care carriers violate qualifying-person", + ), + ), +) +def test_late_transfer_validator_recomputes_live_coupled_constraints( + pool_tool: ModuleType, + target: str, + constraint_column: str, + replacement: object, + error_match: str, +) -> None: + frame, impute, _transition = _authorized_late_impute_fixture( + pool_tool, + _source_frame(), + ) + receipt = impute["stacked_post_puf_transfer"] + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="live coupled-constraint control", + frame=frame, + ) + tables = {entity: frame.table(entity).copy(deep=True) for entity in frame.entities} + person = tables["person"] + recipient = person[support_channel_column("person")].astype(str).eq("acs") + carrier = recipient & pd.to_numeric(person[target], errors="raise").gt(0.0) + assert carrier.any() + person.loc[person.index[carrier][0], constraint_column] = replacement + tables.update({name: frame.link(name) for name in frame.links}) + corrupted = Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + with pytest.raises(ValueError, match=error_match): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + receipt, + boundary="live coupled-constraint mutation", + frame=corrupted, + ) + + def test_stacked_publication_rejects_forged_derived_order_before_any_write( pool_tool: ModuleType, tmp_path: Path, @@ -2804,7 +3231,7 @@ def identity() -> dict[str, object]: current = identity() pool_code = current["pool_code"] assert current["materializer_version"] == 11 - assert current["stacked_authority"]["version"] == 10 + assert current["stacked_authority"]["version"] == 11 assert pool_code["operator_order"] == [ "assemble_stacked_spine", "prepare_multispine_source_inputs_for_clone", @@ -2921,6 +3348,19 @@ def identity() -> dict[str, object]: lambda: late_schedule, ) stale_late_schedule = identity() + with monkeypatch.context() as changed: + calibration_authority = copy.deepcopy( + pool_tool.stacked_spine_authority_receipt() + ) + calibration_authority["components"]["post_transfer_calibration"]["identity"][ + "scope" + ]["reference"] = "forged_reference_scope" + changed.setattr( + pool_tool, + "stacked_spine_authority_receipt", + lambda: calibration_authority, + ) + stale_post_transfer_calibration = identity() with monkeypatch.context() as changed: source_stage_binding = stacked_spine_module._late_source_stage_spec_binding @@ -2953,10 +3393,11 @@ def changed_source_stage_binding( stale_remaining_manifest, stale_tail_contract, stale_late_schedule, + stale_post_transfer_calibration, stale_source_asset, ) } - assert len(digests) == 9 + assert len(digests) == 10 # Positive control: discovery accepts the exact current semantic identity # under the same fixture engine version used to construct it. @@ -4040,6 +4481,33 @@ def tracked_replace(source: Path, destination: Path) -> None: ] +def test_json_ready_omits_empty_opt_in_transfer_pattern_regimes( + pool_tool: ModuleType, +) -> None: + pattern = acs_transfer_module.AcsTransferPattern( + name="fixture", + observed_optional_predictors=(), + predictors=("age",), + seed=1, + weight_kind="source", + donor_rows=2, + recipient_rows=1, + ) + + legacy = pool_tool._json_ready(pattern) + selected = pool_tool._json_ready( + replace( + pattern, + target_regimes=(("fixture_target", "positive_only"),), + ) + ) + + assert isinstance(legacy, dict) + assert "target_regimes" not in legacy + assert isinstance(selected, dict) + assert selected["target_regimes"] == [["fixture_target", "positive_only"]] + + def test_pool_imputation_wires_post_clone_source_chain_after_primary_and_tail( pool_tool: ModuleType, ) -> None: diff --git a/packages/microcosm-build/tests/test_us_post_transfer_calibration.py b/packages/microcosm-build/tests/test_us_post_transfer_calibration.py new file mode 100644 index 00000000..7ffd77a8 --- /dev/null +++ b/packages/microcosm-build/tests/test_us_post_transfer_calibration.py @@ -0,0 +1,1195 @@ +"""Focused contracts for US post-transfer two-part calibration.""" + +from __future__ import annotations + +import hashlib +import json + +import numpy as np +import pandas as pd +import pytest + +from microcosm.build.us_runtime.post_transfer_calibration import ( + POST_TRANSFER_CALIBRATION_SPECS, + PostTransferCalibrationSpec, + apply_post_transfer_calibration, + calibrate_post_transfer_values, + post_transfer_calibration_policy_identity, + post_transfer_calibration_spec, + validate_post_transfer_calibration_receipt, +) +from microcosm.frame import US_SCHEMA, Frame, WeightKind, Weights + + +def _mask(size: int, *positions: int) -> np.ndarray: + result = np.zeros(size, dtype=bool) + result[list(positions)] = True + return result + + +def _match_spec(): + return post_transfer_calibration_spec( + entity="person", + family="source_operator_child_support", + target="child_support_expense", + ) + + +def _preserve_spec(): + return post_transfer_calibration_spec( + entity="person", + family="model_required_numeric", + target="unemployment_compensation", + ) + + +_LATE_MATCH_SPECS = tuple( + spec + for spec in POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "late_transfer" and spec.carrier_mode == "match_reference" +) + + +def _canonical_sha256(value: object) -> str: + return hashlib.sha256( + json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + + +def _assert_canonical_receipt(receipt: dict[str, object]) -> None: + payload = dict(receipt) + observed = payload.pop("sha256") + assert observed == _canonical_sha256(payload) + json.dumps(receipt, allow_nan=False) + + +def test_policy_identity_binds_exact_nine_specs_and_its_content_hash() -> None: + identity = post_transfer_calibration_policy_identity() + payload = dict(identity) + observed_sha256 = payload.pop("sha256") + + assert observed_sha256 == _canonical_sha256(payload) + assert len(POST_TRANSFER_CALIBRATION_SPECS) == 9 + assert [ + f"{target['entity']}/{target['family']}/{target['target']}" + for target in identity["targets"] + ] == sorted(POST_TRANSFER_CALIBRATION_SPECS) + assert { + spec.key + for spec in POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "early_gap_fill" + } == { + "person/model_required_numeric/unemployment_compensation", + ("person/source_operator_prior_year_income/self_employment_income_last_year"), + } + assert { + spec.key + for spec in POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.carrier_mode == "preserve_recipient" + } == { + "person/model_required_numeric/unemployment_compensation", + "person/source_operator_disability_benefits/disability_benefits", + } + + +def test_kernel_and_validator_reject_caller_constructed_undeclared_spec() -> None: + declared = _match_spec() + rogue = PostTransferCalibrationSpec( + entity=declared.entity, + family=declared.family, + target=declared.target, + stage=declared.stage, + carrier_mode="preserve_recipient", + ) + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + with pytest.raises(ValueError, match="exact live declared policy entry"): + calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=rogue, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + canonical = calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=declared, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + with pytest.raises(ValueError, match="exact live declared policy entry"): + validate_post_transfer_calibration_receipt( + canonical.receipt, + spec=rogue, + boundary="rogue spec validator regression", + ) + + +@pytest.mark.parametrize( + "spec", + ( + post_transfer_calibration_spec( + entity="person", + family="adult_care", + target="pre_subsidy_care_expenses", + ), + post_transfer_calibration_spec( + entity="person", + family="source_operator_weeks_unemployed", + target="weeks_unemployed", + ), + ), +) +@pytest.mark.parametrize( + "supply_allowed,supply_additions", ((False, False), (True, False), (False, True)) +) +def test_special_constraint_masks_are_mandatory( + spec: PostTransferCalibrationSpec, + supply_allowed: bool, + supply_additions: bool, +) -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + with pytest.raises(ValueError, match="requires explicit allowed_carrier_rows"): + calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient if supply_allowed else None, + addition_candidate_rows=recipient if supply_additions else None, + ) + + +def test_match_reference_down_rake_keeps_strongest_weighted_prefix() -> None: + values = np.asarray([0.0, 10.0, 100.0, 20.0, 50.0, 100.0, 0.0]) + weights = np.asarray([6.0, 2.0, 2.0, 1.0, 4.0, 5.0, 10.0]) + entity_ids = np.asarray([1, 2, 3, 10, 11, 12, 13]) + reference = _mask(len(values), 0, 1, 2) + recipient = ~reference + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + carriers = set(entity_ids[recipient & (result.values > 0.0)].tolist()) + assert carriers == {11, 12} + assert set(result.values[recipient & (result.values > 0.0)]) <= {10.0, 100.0} + carrier = result.receipt["carrier"] + assert { + key: carrier[key] + for key in ( + "mode", + "reference_positive_mass", + "reference_positive_share", + "target_positive_mass", + "before_positive_mass", + "before_positive_share", + "after_positive_mass", + "after_positive_share", + "residual_after_minus_target", + "absolute_residual", + "removed_rows", + "added_rows", + "disallowed_cleared_rows", + "capacity_limited", + ) + } == { + "mode": "match_reference", + "reference_positive_mass": 4.0, + "reference_positive_share": 0.4, + "target_positive_mass": 8.0, + "before_positive_mass": 10.0, + "before_positive_share": 0.5, + "after_positive_mass": 9.0, + "after_positive_share": 0.45, + "residual_after_minus_target": 1.0, + "absolute_residual": 1.0, + "removed_rows": 1, + "added_rows": 0, + "disallowed_cleared_rows": 0, + "capacity_limited": False, + } + assert carrier["capacity"] == { + "fixed_positive_rows": 0, + "fixed_positive_mass": 0.0, + "allowed_positive_rows_before": 3, + "allowed_positive_mass_before": 10.0, + "addition_candidate_rows": 1, + "addition_candidate_mass": 10.0, + "minimum_attainable_mass": 0.0, + "maximum_attainable_mass": 20.0, + "target_within_attainable_interval": True, + "capacity_boundary_saturated": True, + } + assert carrier["selection"]["action"] == "retain_positive_prefix" + assert carrier["selection"]["chosen_prefix_mass"] == 9.0 + _assert_canonical_receipt(result.receipt) + + +def test_match_reference_equal_distance_uses_lower_mass_and_entity_id_tie() -> None: + # Reference incidence is 0.4 and recipient mass is 10, so the target is 4. + # Equal-valued carriers have weights 3 and 2. Their prefixes 3 and 5 are + # equally distant; lower mass wins, and stable entity id 10 ranks first. + values = np.asarray([0.0, 100.0, 100.0, 100.0, 0.0]) + weights = np.asarray([6.0, 4.0, 2.0, 3.0, 5.0]) + entity_ids = np.asarray([1, 2, 20, 10, 30]) + reference = _mask(len(values), 0, 1) + recipient = ~reference + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + assert set(entity_ids[recipient & (result.values > 0.0)]) == {10} + assert result.receipt["carrier"]["after_positive_mass"] == 3.0 + assert result.receipt["carrier"]["residual_after_minus_target"] == -1.0 + + +def _up_rake_fixture() -> tuple[ + np.ndarray, + np.ndarray, + np.ndarray, + np.ndarray, + np.ndarray, +]: + values = np.asarray([0.0, 100.0, 50.0, 0.0, 0.0, 0.0]) + weights = np.asarray([4.0, 6.0, 2.0, 4.0, 1.0, 3.0]) + entity_ids = np.asarray([1, 2, 40, 30, 10, 20]) + reference = _mask(len(values), 0, 1) + recipient = ~reference + return values, weights, entity_ids, reference, recipient + + +def test_match_reference_up_rake_adds_weighted_prefix_in_entity_id_order() -> None: + values, weights, entity_ids, reference, recipient = _up_rake_fixture() + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + assert set(entity_ids[recipient & (result.values > 0.0)]) == {10, 20, 40} + assert result.receipt["carrier"]["target_positive_mass"] == 6.0 + assert result.receipt["carrier"]["after_positive_mass"] == 6.0 + assert result.receipt["carrier"]["added_rows"] == 2 + assert result.receipt["carrier"]["absolute_residual"] == 0.0 + + +def test_selection_and_amounts_are_invariant_to_input_row_order() -> None: + values, weights, entity_ids, reference, recipient = _up_rake_fixture() + baseline = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + permutation = np.asarray([4, 1, 5, 0, 3, 2]) + shuffled = calibrate_post_transfer_values( + values[permutation], + weights[permutation], + entity_ids[permutation], + spec=_match_spec(), + reference_rows=reference[permutation], + recipient_rows=recipient[permutation], + mutable_rows=recipient[permutation], + ) + + baseline_by_id = dict( + zip(entity_ids.tolist(), baseline.values.tolist(), strict=True) + ) + shuffled_by_id = dict( + zip( + entity_ids[permutation].tolist(), + shuffled.values.tolist(), + strict=True, + ) + ) + assert shuffled_by_id == baseline_by_id + assert shuffled.receipt["carrier"] == baseline.receipt["carrier"] + assert ( + shuffled.receipt["amount"]["anchor_rows"] + == (baseline.receipt["amount"]["anchor_rows"]) + ) + + +def test_preserve_mode_keeps_carriers_and_exactly_anchors_five_quantiles() -> None: + # Donor CDF knots are exactly p10/p25/p50/p75/p100. Recipient upper-CDF + # ranks are p20/p40/p60/p80/p100, so a plain upper-CDF map would start at + # 20 rather than 10. The explicit five battery anchors restore all knots. + values = np.asarray( + [10.0, 20.0, 30.0, 40.0, 50.0, 100.0, 200.0, 300.0, 400.0, 500.0] + ) + weights = np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0]) + entity_ids = np.arange(1, len(values) + 1) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + before_carriers = values[recipient] > 0.0 + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + np.testing.assert_array_equal(result.values[recipient] > 0.0, before_carriers) + np.testing.assert_array_equal( + result.values[recipient], + np.asarray([10.0, 20.0, 30.0, 40.0, 50.0]), + ) + amount = result.receipt["amount"] + assert amount["reference_quantiles"] == [10.0, 20.0, 30.0, 40.0, 50.0] + assert amount["recipient_after_quantiles"] == amount["reference_quantiles"] + assert amount["qed_after"] == 0.0 + assert amount["exact_anchor_count"] == 5 + assert amount["anchor_conflicts"] == [] + assert amount["unanchored_quantiles"] == [] + assert set(result.values[recipient]) <= set(values[reference]) + assert result.receipt["invariants"]["preserve_carriers"] is True + + +def test_negative_negative_zero_zero_weight_and_immutable_bytes_are_exact() -> None: + values = np.asarray( + [0.0, 10.0, 20.0, -7.25, -0.0, 999.0, 123.0, 50.0, 0.0], + dtype=np.float64, + ) + weights = np.asarray([1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 1.0, 1.0, 2.0]) + entity_ids = np.arange(1, len(values) + 1) + reference = _mask(len(values), 0, 1, 2) + recipient = ~reference + mutable = recipient.copy() + mutable[6] = False + byte_exact = _mask(len(values), 3, 4, 5, 6) + before_bits = values.view(np.uint64).copy() + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=post_transfer_calibration_spec( + entity="person", + family="source_operator_prior_year_income", + target="self_employment_income_last_year", + ), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + ) + + np.testing.assert_array_equal( + result.values.view(np.uint64)[byte_exact], before_bits[byte_exact] + ) + assert np.signbit(result.values[4]) + mapped = mutable & (weights > 0.0) & (result.values > 0.0) + assert set(result.values[mapped]) <= {10.0, 20.0} + assert result.receipt["carrier"]["after_positive_mass"] == 4.0 + assert result.receipt["invariants"] == { + "immutable_bytes_preserved": True, + "negative_bytes_preserved": True, + "negative_zero_bytes_preserved": True, + "zero_weight_bytes_preserved": True, + "preserve_carriers": False, + "allowed_carrier_violations": 0, + "exact_quantile_anchors": False, + } + + +def test_float32_values_fail_closed_before_byte_preservation_is_claimed() -> None: + values = np.asarray([0.0, 10.0, 20.0, 30.0], dtype=np.float32) + reference = _mask(len(values), 0, 1) + recipient = ~reference + + with pytest.raises(ValueError, match="exact float64 dtype"): + calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(1, len(values) + 1), + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + +def test_allowed_masks_clear_forbidden_carrier_and_report_capacity_shortfall() -> None: + values = np.asarray([0.0, 10.0, 100.0, 300.0, 0.0, 0.0]) + weights = np.asarray([2.0, 8.0, 1.0, 3.0, 2.0, 4.0]) + entity_ids = np.arange(1, len(values) + 1) + reference = _mask(len(values), 0, 1) + recipient = ~reference + allowed = _mask(len(values), 2, 4) + additions = _mask(len(values), 4) + + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=post_transfer_calibration_spec( + entity="person", + family="source_operator_weeks_unemployed", + target="weeks_unemployed", + ), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=allowed, + addition_candidate_rows=additions, + ) + + assert set(entity_ids[recipient & (result.values > 0.0)]) == {3, 5} + carrier = result.receipt["carrier"] + assert carrier["target_positive_mass"] == 8.0 + assert carrier["after_positive_mass"] == 3.0 + assert carrier["residual_after_minus_target"] == -5.0 + assert carrier["disallowed_cleared_rows"] == 1 + assert carrier["added_rows"] == 1 + assert carrier["capacity_limited"] is True + assert carrier["capacity"] == { + "fixed_positive_rows": 0, + "fixed_positive_mass": 0.0, + "allowed_positive_rows_before": 1, + "allowed_positive_mass_before": 1.0, + "addition_candidate_rows": 1, + "addition_candidate_mass": 2.0, + "minimum_attainable_mass": 0.0, + "maximum_attainable_mass": 3.0, + "target_within_attainable_interval": False, + "capacity_boundary_saturated": True, + } + assert carrier["selection"]["action"] == "add_zero_prefix" + assert carrier["selection"]["chosen_prefix_mass"] == 2.0 + assert result.receipt["invariants"]["allowed_carrier_violations"] == 0 + + scope = result.receipt["scope"] + assert ( + scope["allowed_carrier_rows_sha256"] + == hashlib.sha256(np.ascontiguousarray(allowed).tobytes(order="C")).hexdigest() + ) + assert ( + scope["addition_candidate_rows_sha256"] + == hashlib.sha256( + np.ascontiguousarray(additions).tobytes(order="C") + ).hexdigest() + ) + validate_post_transfer_calibration_receipt( + result.receipt, + spec=post_transfer_calibration_spec( + entity="person", + family="source_operator_weeks_unemployed", + target="weeks_unemployed", + ), + boundary="allowed-mask binding regression", + ) + + stripped = { + **result.receipt, + "scope": dict(result.receipt["scope"]), + } + stripped["scope"].pop("addition_candidate_rows_sha256") + stripped_payload = dict(stripped) + stripped_payload.pop("sha256") + stripped["sha256"] = _canonical_sha256(stripped_payload) + with pytest.raises(ValueError, match="scope evidence is incomplete"): + validate_post_transfer_calibration_receipt( + stripped, + spec=post_transfer_calibration_spec( + entity="person", + family="source_operator_weeks_unemployed", + target="weeks_unemployed", + ), + boundary="stripped addition-mask binding regression", + ) + + +@pytest.mark.parametrize( + "spec", + _LATE_MATCH_SPECS, + ids=lambda spec: spec.target, +) +def test_late_match_receipts_share_production_candidate_prefix_mass( + spec: PostTransferCalibrationSpec, +) -> None: + production_weight_hex = ( + "0x1.3733a66e776b1p+11", + "0x1.67143630b12ccp+8", + "0x1.313794763032ep+12", + "0x1.eabdc186d0017p+10", + "0x1.67143630b12ccp+9", + "0x1.badd31c4963fcp+9", + "0x1.d8c98b9dfa58ep+11", + "0x1.f6b5e5775e71ep+8", + "0x1.253f7085a1c27p+11", + "0x1.fcb1f76fa5aa1p+11", + "0x1.b7df28c872a3bp+11", + "0x1.d2cd79a5b320ap+10", + "0x1.d8c98b9dfa58ep+10", + "0x1.104d31a0a87dbp+13", + "0x1.04550db01a0d4p+11", + "0x1.104d31a0a87dbp+11", + "0x1.c6d555b524b03p+10", + "0x1.e640b40c9a975p+12", + "0x1.dec59d9641911p+8", + "0x1.04550db01a0d4p+12", + "0x1.5b1c124022bc6p+11", + "0x1.5c9b16be348a6p+12", + "0x1.96f4c5f2eaee8p+10", + "0x1.730c5a213f9d3p+9", + "0x1.14ca3f1adde7dp+12", + "0x1.9cf0d7eb3226bp+10", + "0x1.3d2fb866bea35p+11", + "0x1.6b9143aae696fp+12", + "0x1.1f435e8d5a8a3p+10", + "0x1.e4c1af8e88c94p+10", + "0x1.3733a66e776b1p+9", + "0x1.3733a66e776b1p+8", + ) + candidate_weights = np.asarray( + [float.fromhex(value) for value in production_weight_hex], + dtype=" None: + recipient_weights = np.asarray( + [ + float.fromhex("0x1.b69b69add8605p+10"), + float.fromhex("0x1.ca7a5e98478d8p+11"), + float.fromhex("0x1.630c87822425bp+11"), + float.fromhex("0x1.b5e61d1252c51p+12"), + ], + dtype=np.float64, + ) + whole_mass = float(recipient_weights.sum()) + positive_mass = float(recipient_weights[0]) + zero_candidate_mass = float(np.cumsum(recipient_weights[1:], dtype=np.float64)[-1]) + partition_mass = positive_mass + zero_candidate_mass + assert whole_mass.hex() == "0x1.dd2835457f5b6p+13" + assert partition_mass.hex() == "0x1.dd2835457f5b7p+13" + assert partition_mass > whole_mass + + weights = np.concatenate((np.asarray([1.0, 1.0]), recipient_weights)) + values = np.asarray([1.0, 0.0, 4.0, 0.0, 0.0, 0.0], dtype=np.float64) + reference = _mask(len(values), 0, 1) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + weights, + np.arange(len(values)), + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + allowed_carrier_rows=recipient, + addition_candidate_rows=recipient, + ) + + capacity = result.receipt["carrier"]["capacity"] + assert result.receipt["weights"]["recipient_total"] == whole_mass + assert capacity["allowed_positive_mass_before"] == positive_mass + assert capacity["addition_candidate_mass"] == zero_candidate_mass + assert capacity["maximum_attainable_mass"] == whole_mass + assert ( + capacity["allowed_positive_mass_before"] + capacity["addition_candidate_mass"] + > capacity["maximum_attainable_mass"] + ) + validate_post_transfer_calibration_receipt( + result.receipt, + spec=spec, + boundary="whole attainable-row union regression", + ) + + +@pytest.mark.parametrize( + "spec", + _LATE_MATCH_SPECS, + ids=lambda spec: spec.target, +) +def test_late_match_capacity_preserves_exact_subset_bound( + spec: PostTransferCalibrationSpec, +) -> None: + recipient_weights = np.asarray( + [ + float.fromhex("0x1.5cedcaec2f7f6p+5"), + float.fromhex("0x1.238d33e1a2a4ep-6"), + float.fromhex("0x1.dfbe6877a2824p-9"), + float.fromhex("0x1.7b846b4340ddfp+33"), + float.fromhex("0x1.c458051dac358p+31"), + float.fromhex("0x1.43314c2e4a93ep+48"), + float.fromhex("0x1.1d1111fbd2c7fp+21"), + float.fromhex("0x1.751b9d46fb62fp+24"), + ], + dtype=np.float64, + ) + attainable_in_recipient = np.asarray( + [True, True, False, True, True, True, True, True], + dtype=bool, + ) + recipient_total = float(recipient_weights.sum()) + regrouped_subset_mass = float(recipient_weights[attainable_in_recipient].sum()) + assert recipient_total.hex() == "0x1.433526fbe1945p+48" + assert regrouped_subset_mass.hex() == "0x1.433526fbe1946p+48" + assert regrouped_subset_mass > recipient_total + + weights = np.concatenate((np.asarray([1.0, 1.0]), recipient_weights)) + values = np.zeros(len(weights), dtype=np.float64) + values[0] = 1.0 + values[7] = 4.0 + reference = _mask(len(values), 0, 1) + recipient = ~reference + mutable = np.concatenate((np.asarray([False, False]), attainable_in_recipient)) + result = calibrate_post_transfer_values( + values, + weights, + np.arange(len(values)), + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + allowed_carrier_rows=mutable, + addition_candidate_rows=mutable, + ) + + capacity = result.receipt["carrier"]["capacity"] + assert result.receipt["weights"]["recipient_total"] == recipient_total + assert capacity["maximum_attainable_mass"] == recipient_total + validate_post_transfer_calibration_receipt( + result.receipt, + spec=spec, + boundary="exact attainable-subset bound regression", + ) + + +def test_retain_capacity_and_selection_share_one_ordered_prefix_mass() -> None: + candidate_weights = np.asarray( + [ + 7058.927001418665, + 7659.729944427913, + 2574.5891324792306, + 7988.62015511553, + 1693.9976601157439, + 5237.851828796898, + 5414.693471133362, + 8727.20414048527, + ], + dtype=np.float64, + ) + masked_mass = float(candidate_weights.sum()) + ordered_prefix_mass = float(np.cumsum(candidate_weights, dtype=np.float64)[-1]) + assert masked_mass.hex() == "0x1.6a273a06e913cp+15" + assert ordered_prefix_mass.hex() == "0x1.6a273a06e913dp+15" + + # A 50% reference share requests the masked total of every mutable + # positive. Descending values preserve the declared candidate order, while + # the immutable zero recipient keeps capacity below total recipient mass. + weights = np.concatenate( + (np.asarray([1.0, 1.0]), candidate_weights, np.asarray([masked_mass])) + ) + values = np.concatenate( + ( + np.asarray([1.0, 0.0]), + np.arange(len(candidate_weights), 0, -1, dtype=np.float64), + np.asarray([0.0]), + ) + ) + reference = _mask(len(values), 0, 1) + recipient = ~reference + mutable = np.zeros(len(values), dtype=bool) + mutable[2:10] = True + + result = calibrate_post_transfer_values( + values, + weights, + np.arange(len(values)), + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + ) + + capacity = result.receipt["carrier"]["capacity"] + selection = result.receipt["carrier"]["selection"] + assert selection["action"] == "retain_positive_prefix" + assert capacity["allowed_positive_mass_before"] == ordered_prefix_mass + assert selection["candidate_mass"] == ordered_prefix_mass + assert selection["chosen_prefix_mass"] == ordered_prefix_mass + assert selection["upper_prefix_mass"] == ordered_prefix_mass + validate_post_transfer_calibration_receipt( + result.receipt, + spec=_match_spec(), + boundary="retain production-weight reduction regression", + ) + + +def test_match_reference_proves_immutable_positive_floor_saturation() -> None: + values = np.asarray([0.0, 10.0, 10.0, 10.0, 10.0, 10.0, 20.0, 20.0, 20.0, 20.0]) + weights = np.asarray([8.0, 0.5, 0.5, 0.5, 0.5] * 2) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + mutable = _mask(len(values), 6, 7, 8, 9) + + result = calibrate_post_transfer_values( + values, + weights, + np.arange(len(values)), + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + ) + + carrier = result.receipt["carrier"] + assert carrier["target_positive_mass"] == 2.0 + assert carrier["after_positive_mass"] == 8.0 + assert carrier["capacity_limited"] is True + assert carrier["capacity"]["minimum_attainable_mass"] == 8.0 + assert carrier["capacity"]["maximum_attainable_mass"] == 10.0 + assert carrier["capacity"]["capacity_boundary_saturated"] is True + assert carrier["selection"]["action"] == "retain_positive_prefix" + assert carrier["selection"]["chosen_prefix_mass"] == 0.0 + assert result.values[5] == 10.0 + assert np.all(result.values[6:] == 0.0) + validate_post_transfer_calibration_receipt( + result.receipt, + spec=_match_spec(), + boundary="immutable positive floor saturation", + ) + + +def test_validator_rejects_rehashed_self_consistent_zero_carrier_forgery() -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + forged = {**result.receipt, "carrier": dict(result.receipt["carrier"])} + for key in ( + "reference_positive_mass", + "reference_positive_share", + "target_positive_mass", + "before_positive_mass", + "before_positive_share", + "after_positive_mass", + "after_positive_share", + "residual_after_minus_target", + "absolute_residual", + ): + forged["carrier"][key] = 0.0 + payload = dict(forged) + payload.pop("sha256") + forged["sha256"] = _canonical_sha256(payload) + + with pytest.raises(ValueError, match="carrier relationships are invalid"): + validate_post_transfer_calibration_receipt( + forged, + spec=_preserve_spec(), + boundary="zero carrier forgery", + ) + + +def test_validator_rejects_rehashed_stripped_amount_evidence() -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + forged = { + **result.receipt, + "amount": { + key: result.receipt["amount"][key] + for key in ( + "donor_support_violations", + "status", + "exact_anchor_count", + "anchor_conflicts", + "unanchored_quantiles", + ) + }, + } + payload = dict(forged) + payload.pop("sha256") + forged["sha256"] = _canonical_sha256(payload) + + with pytest.raises(ValueError, match="amount schema is invalid"): + validate_post_transfer_calibration_receipt( + forged, + spec=_preserve_spec(), + boundary="stripped amount forgery", + ) + + +def test_validator_rejects_rehashed_verification_contract_tampering() -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + forged = { + **result.receipt, + "verification_contract": dict(result.receipt["verification_contract"]), + } + forged["verification_contract"]["terminal_pre_state_replay"] = True + payload = dict(forged) + payload.pop("sha256") + forged["sha256"] = _canonical_sha256(payload) + + with pytest.raises(ValueError, match="policy/spec binding is invalid"): + validate_post_transfer_calibration_receipt( + forged, + spec=_preserve_spec(), + boundary="forged verification boundary", + ) + + +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ("carrier", "carrier evidence is incomplete"), + ("weights", "weight evidence is absent"), + ("scope_count", "scope evidence is incomplete"), + ), +) +def test_validator_rejects_rehashed_stripped_core_evidence( + mutation: str, + error_match: str, +) -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0]), + np.arange(len(values)), + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + stripped = {**result.receipt, "scope": dict(result.receipt["scope"])} + if mutation == "scope_count": + stripped["scope"].pop("reference_rows") + else: + stripped.pop(mutation) + payload = dict(stripped) + payload.pop("sha256") + stripped["sha256"] = _canonical_sha256(payload) + + with pytest.raises(ValueError, match=error_match): + validate_post_transfer_calibration_receipt( + stripped, + spec=_preserve_spec(), + boundary=f"stripped {mutation} regression", + ) + + +def test_validator_rejects_same_count_different_mask_context_transplant() -> None: + spec = _preserve_spec() + values_a = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + values_b = np.asarray([11.0, 1.0, 21.0, 2.0, 31.0, 3.0, 41.0, 4.0, 51.0, 5.0]) + reference_a = _mask(len(values_a), 0, 1, 2, 3, 4) + reference_b = _mask(len(values_b), 0, 2, 4, 6, 8) + recipient_a = ~reference_a + recipient_b = ~reference_b + result_a = calibrate_post_transfer_values( + values_a, + np.ones(len(values_a), dtype=np.float64), + np.arange(len(values_a)), + spec=spec, + reference_rows=reference_a, + recipient_rows=recipient_a, + mutable_rows=recipient_a, + ) + result_b = calibrate_post_transfer_values( + values_b, + np.full(len(values_b), 2.0, dtype=np.float64), + np.arange(100, 100 + len(values_b)), + spec=spec, + reference_rows=reference_b, + recipient_rows=recipient_b, + mutable_rows=recipient_b, + ) + + for count_key in ("reference_rows", "recipient_rows", "mutable_rows"): + assert ( + result_a.receipt["scope"][count_key] + == (result_b.receipt["scope"][count_key]) + ) + expected_mask_scope = { + key: result_b.receipt["scope"][key] + for key in ( + "reference_rows_sha256", + "recipient_rows_sha256", + "mutable_rows_sha256", + ) + } + with pytest.raises(ValueError, match="scope does not match.*live context"): + validate_post_transfer_calibration_receipt( + result_a.receipt, + spec=spec, + boundary="same-count context transplant", + expected_scope=expected_mask_scope, + ) + with pytest.raises(ValueError, match="weights do not match.*live context"): + validate_post_transfer_calibration_receipt( + result_a.receipt, + spec=spec, + boundary="weight context transplant", + expected_scope=result_a.receipt["scope"], + expected_weights_sha256=result_b.receipt["weights"]["sha256"], + ) + + +def test_validator_rejects_rehashed_forged_scope_against_expected_context() -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 1.0, 2.0, 3.0, 4.0, 5.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + result = calibrate_post_transfer_values( + values, + np.ones(len(values), dtype=np.float64), + np.arange(len(values)), + spec=_preserve_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + expected_scope = dict(result.receipt["scope"]) + forged = {**result.receipt, "scope": dict(result.receipt["scope"])} + forged["scope"]["reference_rows_sha256"] = "0" * 64 + payload = dict(forged) + payload.pop("sha256") + forged["sha256"] = _canonical_sha256(payload) + + with pytest.raises(ValueError, match="scope does not match.*live context"): + validate_post_transfer_calibration_receipt( + forged, + spec=_preserve_spec(), + boundary="rehashed forged scope", + expected_scope=expected_scope, + expected_weights_sha256=result.receipt["weights"]["sha256"], + ) + + +def test_sparse_full_recipient_cdf_marks_exact_anchors_infeasible() -> None: + values = np.asarray([10.0, 20.0, 30.0, 40.0, 50.0, 999.0]) + weights = np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 20.0]) + reference = _mask(len(values), 0, 1, 2, 3, 4) + recipient = ~reference + + result = calibrate_post_transfer_values( + values, + weights, + np.arange(1, len(values) + 1), + spec=_match_spec(), + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + assert result.receipt["amount"]["status"] == "infeasible_exact_anchors" + assert result.receipt["amount"]["exact_anchor_count"] < 5 + assert result.receipt["amount"]["anchor_conflicts"] + assert result.receipt["invariants"]["exact_quantile_anchors"] is False + + +def _frame_without_support_provenance() -> Frame: + ids = np.arange(1, 5, dtype=np.int64) + person = pd.DataFrame( + { + "person_id": ids + 100, + "person_household_id": ids, + "person_tax_unit_id": ids + 10, + "person_spm_unit_id": ids + 20, + "person_family_id": ids + 30, + "person_marital_unit_id": ids + 40, + } + ) + tables = { + "person": person, + "household": pd.DataFrame({"household_id": ids}), + "tax_unit": pd.DataFrame({"tax_unit_id": ids + 10}), + "spm_unit": pd.DataFrame( + { + "spm_unit_id": ids + 20, + "spm_unit_energy_subsidy": [0.0, 10.0, 100.0, 200.0], + } + ), + "family": pd.DataFrame({"family_id": ids + 30}), + "marital_unit": pd.DataFrame({"marital_unit_id": ids + 40}), + } + return Frame( + tables, + US_SCHEMA, + { + "household": Weights( + np.asarray([6.0, 4.0, 1.0, 3.0]), + WeightKind.DESIGN, + ) + }, + metadata={"fixture": {"provenance_blind": True}}, + ) + + +def test_frame_wrapper_uses_resolved_entity_weights_without_provenance_columns() -> ( + None +): + frame = _frame_without_support_provenance() + spm_before = frame.table("spm_unit").copy(deep=True) + reference = _mask(4, 0, 1) + recipient = _mask(4, 2, 3) + + result = apply_post_transfer_calibration( + frame, + entity="spm_unit", + family="source_operator_energy_subsidy", + target="spm_unit_energy_subsidy", + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + + assert not any("support_" in column for column in frame.table("spm_unit")) + assert result.receipt["weights"] == { + "sha256": result.receipt["weights"]["sha256"], + "reference_total": 10.0, + "recipient_total": 4.0, + "kind": WeightKind.DESIGN.value, + } + output_spm = result.frame.table("spm_unit") + assert output_spm["spm_unit_energy_subsidy"].tolist() == [0.0, 10.0, 0.0, 10.0] + pd.testing.assert_frame_equal(frame.table("spm_unit"), spm_before) + for entity in ("person", "household", "tax_unit", "family", "marital_unit"): + pd.testing.assert_frame_equal(result.frame.table(entity), frame.table(entity)) + assert result.frame.metadata == frame.metadata + assert result.frame.mass_log == frame.mass_log + _assert_canonical_receipt(result.receipt) + + +def test_frame_helper_rejects_sparse_infeasible_exact_anchors() -> None: + frame = _frame_without_support_provenance() + reference = _mask(4, 1, 2) + recipient = _mask(4, 3) + + with pytest.raises(ValueError, match=r"exact quantile anchors.*infeasible"): + apply_post_transfer_calibration( + frame, + entity="spm_unit", + family="source_operator_energy_subsidy", + target="spm_unit_energy_subsidy", + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) diff --git a/packages/microcosm-build/tests/test_us_spec_bundle.py b/packages/microcosm-build/tests/test_us_spec_bundle.py index 71578924..25f2914a 100644 --- a/packages/microcosm-build/tests/test_us_spec_bundle.py +++ b/packages/microcosm-build/tests/test_us_spec_bundle.py @@ -704,7 +704,7 @@ def test_typed_imputation_reconstructs_all_constants_authority_components( ) -def test_authored_imputation_contains_only_external_asset_sha256_pins( +def test_authored_imputation_sha256_fields_are_assets_or_policy_identity( generated_documents: dict[str, dict[str, object]], ) -> None: imputation = generated_documents["imputation.yaml"] @@ -713,19 +713,38 @@ def test_authored_imputation_contains_only_external_asset_sha256_pins( assert "source_stage_asset" not in graph["resource_semantics"] asset_pins: list[tuple[str, str]] = [] + identity_digests: list[tuple[tuple[str, ...], str]] = [] - def collect_sha256(value: object) -> None: + def collect_sha256(value: object, path: tuple[str, ...] = ()) -> None: if isinstance(value, dict): for key, child in value.items(): + child_path = (*path, key) if key.endswith("sha256"): - assert key == "asset_sha256" - asset_pins.append((str(value["asset"]), str(child))) - collect_sha256(child) + if key == "asset_sha256": + asset_pins.append((str(value["asset"]), str(child))) + else: + identity_digests.append((child_path, str(child))) + collect_sha256(child, child_path) elif isinstance(value, list): for child in value: - collect_sha256(child) + collect_sha256(child, (*path, "[]")) collect_sha256(imputation) + assert identity_digests == [ + ( + ( + "models", + "regime_gated_qrf", + "post_draw_calibration", + "sha256", + ), + str( + imputation["models"]["regime_gated_qrf"]["post_draw_calibration"][ + "sha256" + ] + ), + ) + ] assert len(asset_pins) == 2 assert set(asset_pins) == { ( diff --git a/packages/microcosm-build/tests/test_us_spine_blindness.py b/packages/microcosm-build/tests/test_us_spine_blindness.py index 9267ed80..692b9913 100644 --- a/packages/microcosm-build/tests/test_us_spine_blindness.py +++ b/packages/microcosm-build/tests/test_us_spine_blindness.py @@ -159,6 +159,7 @@ "medicare_take_up.py", "multispine_pool.py", "other_health_insurance.py", + "post_transfer_calibration.py", "prior_year_income.py", "qbi_inputs.py", "retirement_contributions.py", @@ -3278,8 +3279,8 @@ def test_pool_build_tool_import_graph_is_source_spine_blind() -> None: for tool in _SPINE_BLIND_BUILD_TOOLS: runtime_graph, missing_modules = _us_runtime_import_graph(tool) - assert len(runtime_graph) == 65, ( - f"{tool.name} must reach the pinned 65-module runtime graph; " + assert len(runtime_graph) == 66, ( + f"{tool.name} must reach the pinned 66-module runtime graph; " f"reached {len(runtime_graph)}" ) assert not missing_modules, ( diff --git a/packages/microcosm-build/tests/test_us_stacked_spine.py b/packages/microcosm-build/tests/test_us_stacked_spine.py index 4e1b8f9a..e414dfa7 100644 --- a/packages/microcosm-build/tests/test_us_stacked_spine.py +++ b/packages/microcosm-build/tests/test_us_stacked_spine.py @@ -27,6 +27,7 @@ import microcosm.build.us_runtime.acs_income_universe as universe_module import microcosm.build.us_runtime.acs_transfer as acs_transfer_module import microcosm.build.us_runtime.multispine_pool as multispine_pool_module +import microcosm.build.us_runtime.post_transfer_calibration as post_transfer_calibration_runtime import microcosm.build.us_runtime.puf_capital_gains_tail as tail_module import microcosm.build.us_runtime.puf_support as puf_support_module import microcosm.build.us_runtime.stacked_spine as stacked_spine_module @@ -39,7 +40,11 @@ from microcosm.build.us_runtime.acs_income_universe import ( apply_acs_pums_earnings_universe_zeros, ) -from microcosm.build.us_runtime.acs_transfer import AcsTransferResult +from microcosm.build.us_runtime.acs_transfer import ( + AcsImputedInput, + AcsTransferPattern, + AcsTransferResult, +) from microcosm.build.us_runtime.acs_transfer_bank import AcsTransferTargetBankStore from microcosm.build.us_runtime.late_producer_dag import ( ProducerContract, @@ -1680,6 +1685,17 @@ def test_production_entrypoints_take_no_authority_parameters() -> None: ) +def test_canonical_gap_fill_rejects_nondefault_target_fit_width() -> None: + with pytest.raises( + ValueError, + match="Canonical stacked gap fill requires max_targets_per_fit=8", + ): + gap_fill_stacked_spine( + _stacked_gap_fixture(), + max_targets_per_fit=1, + ) + + def test_canonical_authority_objects_are_deeply_immutable() -> None: plan = stacked_spine_module.CANONICAL_STACKED_GAP_FILL_PLAN post_puf_surface = stacked_spine_module.CANONICAL_STACKED_POST_PUF_TRANSFER_SURFACE @@ -1693,6 +1709,9 @@ def test_canonical_authority_objects_are_deeply_immutable() -> None: registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_METRIC_REGISTRY joint_registry = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_JOINT_METRIC_REGISTRY profile = stacked_spine_module.CANONICAL_ORIGIN_BATTERY_SUPPORT_PROFILE + calibration = ( + stacked_spine_module._CANONICAL_STACKED_AUTHORITY.post_transfer_calibration + ) assert isinstance(plan, tuple) with pytest.raises(TypeError): @@ -1720,6 +1739,10 @@ def test_canonical_authority_objects_are_deeply_immutable() -> None: ] = "categorical_tvd" with pytest.raises(FrozenInstanceError): profile.min_effective_support = 50 + with pytest.raises(TypeError): + calibration["scope"] = {} + with pytest.raises(TypeError): + calibration["scope"]["reference"] = "forged" def test_canonical_metric_registry_covers_the_declared_131_target_split() -> None: @@ -2515,6 +2538,484 @@ def test_gap_fill_plan_covers_declared_families_exactly() -> None: assert early_keys | late_keys == full_keys +def _canonical_gap_fill_calibration_receipt() -> dict[str, object]: + policy = ( + post_transfer_calibration_runtime.post_transfer_calibration_policy_identity() + ) + early_specs = { + spec.key: spec + for spec in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "early_gap_fill" + } + values = np.asarray( + [10.0, 20.0, 30.0, 40.0, 50.0, 100.0, 200.0, 300.0, 400.0, 500.0] + ) + weights = np.asarray([2.0, 3.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0]) + reference = np.asarray([True] * 5 + [False] * 5) + recipient = ~reference + directions: dict[str, object] = {} + for direction in stacked_spine_module.CANONICAL_STACKED_GAP_FILL_PLAN: + target_keys = { + f"{entity}/{family}/{target}" + for entity, families in direction.target_families.items() + for family, targets in families.items() + for target in targets + } + calibrated_keys = sorted(target_keys & set(early_specs)) + target_receipts: dict[str, dict[str, object]] = { + key: { + "authorized_null_rows": 0, + "imputed_rows": 0, + "unmodeled_rows": 0, + "residual_null_rows": 0, + } + for key in target_keys + } + for key in calibrated_keys: + spec = early_specs[key] + calibration_result = ( + post_transfer_calibration_runtime.calibrate_post_transfer_values( + values, + weights, + np.arange(1, len(values) + 1), + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=recipient, + ) + ) + calibration = calibration_result.receipt + scope = calibration["scope"] + target_receipts[key]["post_transfer_calibration"] = { + "stage": "early_gap_fill", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": scope["reference_rows"], + "recipient_rows": scope["recipient_rows"], + "mutable_rows": scope["mutable_rows"], + "constraint": {"constraint": "none"}, + "context_binding": { + "scope": dict(scope), + "weights_sha256": calibration["weights"]["sha256"], + "live_output": { + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "reference_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + np.arange(1, len(values) + 1)[reference] + ) + ), + "recipient_entity_ids_sha256": ( + stacked_spine_module._post_transfer_entity_ids_sha256( + np.arange(1, len(values) + 1)[recipient] + ) + ), + "reference_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + calibration_result.values[reference], + boundary="synthetic reference calibration output", + ) + ), + "recipient_output_values_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + calibration_result.values[recipient], + boundary="synthetic recipient calibration output", + ) + ), + "reference_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[reference], + boundary="synthetic reference calibration weights", + ) + ), + "recipient_weights_sha256": ( + stacked_spine_module._post_transfer_float64_sha256( + weights[recipient], + boundary="synthetic recipient calibration weights", + ) + ), + }, + }, + "calibration": calibration, + } + directions[direction.name] = { + "targets": target_receipts, + "post_transfer_calibration": { + "policy_sha256": policy["sha256"], + "target_count": len(calibrated_keys), + "targets": calibrated_keys, + }, + } + return { + "authority": stacked_spine_module.stacked_spine_authority_receipt(), + "directions": directions, + } + + +def _canonical_gap_fill_receipt_with_pattern_evidence() -> tuple[ + dict[str, object], + str, + str, + str, + str, + tuple[str, ...], +]: + receipt = _canonical_gap_fill_calibration_receipt() + early_keys = { + spec.key + for spec in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "early_gap_fill" + } + selected: tuple[str, str, str, str, str, tuple[str, ...]] | None = None + for direction in stacked_spine_module.CANONICAL_STACKED_GAP_FILL_PLAN: + for entity, families in direction.target_families.items(): + for family, targets in families.items(): + for target in targets: + key = f"{entity}/{family}/{target}" + if key in early_keys: + selected = ( + direction.name, + key, + entity, + family, + target, + targets, + ) + break + if selected is not None: + break + if selected is not None: + break + if selected is not None: + break + assert selected is not None + direction_name, key, entity, family, target, family_targets = selected + evidence_targets = tuple( + family_target + for family_target in family_targets + if f"{entity}/{family}/{family_target}" in early_keys + ) + model_targets = acs_transfer_module._model_target_names(evidence_targets) + required_predictors, optional_predictors = ( + stacked_spine_module._acs_pattern_predictor_authority( + entity=entity, + family_targets=family_targets, + ) + ) + selected_optional = optional_predictors[:1] + patterns = tuple( + AcsTransferPattern( + name=acs_transfer_module._pattern_name(index, observed_optional), + observed_optional_predictors=observed_optional, + predictors=(*required_predictors, *observed_optional), + seed=index, + weight_kind="design", + donor_rows=1, + recipient_rows=1, + target_regimes=tuple( + (model_target, "positive_only") for model_target in model_targets + ), + ) + for index, observed_optional in enumerate(((), selected_optional)) + ) + record = AcsImputedInput( + column=target, + entity=entity, + family=family, + donor_spine="synthetic_gap_validator_fixture", + donor_channel=None, + predictors=(*required_predictors, *selected_optional), + seed=0, + weight_kind="design", + patterns=patterns, + imputed_recipient_rows=2, + ) + target_receipt = receipt["directions"][direction_name]["targets"][key] + target_receipt.update( + { + "authorized_null_rows": 2, + "imputed_rows": 2, + "unmodeled_rows": 0, + "residual_null_rows": 0, + "qrf_pattern_evidence": ( + stacked_spine_module._acs_imputed_pattern_evidence(record) + ), + } + ) + return receipt, direction_name, key, entity, family, family_targets + + +def test_gap_fill_validator_accepts_canonical_calibration_evidence() -> None: + stacked_spine_module.validate_stacked_gap_fill_receipt( + _canonical_gap_fill_calibration_receipt(), + boundary="canonical early calibration evidence control", + ) + + +def test_gap_fill_qrf_binding_excludes_unassigned_batched_targets() -> None: + receipt = _canonical_gap_fill_calibration_receipt() + direction = next( + item + for item in stacked_spine_module.CANONICAL_STACKED_GAP_FILL_PLAN + if item.name == "asec_survey_to_acs" + ) + family = "puf_tax_itemization" + targets = direction.target_families["person"][family] + target = "taxable_interest_income" + key = f"person/{family}/{target}" + target_receipt = receipt["directions"][direction.name]["targets"][key] + legacy_counts = { + "authorized_null_rows": 1, + "imputed_rows": 1, + "unmodeled_rows": 0, + "residual_null_rows": 0, + } + target_receipt.update(legacy_counts) + + batch_targets = targets[ + : acs_transfer_module.DEFAULT_ACS_TRANSFER_MAX_TARGETS_PER_FIT + ] + required_predictors, _optional_predictors = ( + stacked_spine_module._acs_pattern_predictor_authority( + entity="person", + family_targets=batch_targets, + ) + ) + record = AcsImputedInput( + column=target, + entity="person", + family=f"{family}__batch_1", + donor_spine="synthetic_batched_gap_validator_fixture", + donor_channel=None, + predictors=required_predictors, + seed=0, + weight_kind="design", + patterns=( + AcsTransferPattern( + name=acs_transfer_module._pattern_name(0, ()), + observed_optional_predictors=(), + predictors=required_predictors, + seed=0, + weight_kind="design", + donor_rows=1, + recipient_rows=1, + target_regimes=tuple( + (model_target, "positive_only") + for model_target in acs_transfer_module._model_target_names( + batch_targets + ) + ), + ), + ), + imputed_recipient_rows=1, + ) + target_receipt["qrf_pattern_evidence"] = ( + stacked_spine_module._acs_imputed_pattern_evidence(record) + ) + + with pytest.raises( + ValueError, + match="undeclared ACS QRF pattern evidence.*taxable_interest_income", + ): + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="unassigned batched QRF evidence", + ) + + target_receipt.pop("qrf_pattern_evidence") + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="unassigned legacy target receipt", + ) + assert target_receipt == legacy_counts + + +def test_gap_fill_validator_rejects_unassigned_legacy_count_tampering() -> None: + receipt = _canonical_gap_fill_calibration_receipt() + target_receipt = receipt["directions"]["asec_survey_to_acs"]["targets"][ + "person/puf_tax_itemization/taxable_interest_income" + ] + target_receipt.update( + { + "authorized_null_rows": 0, + "imputed_rows": 1, + "unmodeled_rows": 0, + "residual_null_rows": 99, + } + ) + + with pytest.raises(ValueError, match="ACS transfer row-count"): + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="forged unassigned early transfer counts", + ) + + +def test_gap_fill_validator_rejects_unassigned_legacy_count_stripping() -> None: + receipt = _canonical_gap_fill_calibration_receipt() + target_receipt = receipt["directions"]["asec_survey_to_acs"]["targets"][ + "person/puf_tax_itemization/taxable_interest_income" + ] + for field in stacked_spine_module._ACS_TRANSFER_ROW_COUNT_FIELDS: + target_receipt.pop(field) + + with pytest.raises(ValueError, match="ACS transfer row-count schema is invalid"): + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="stripped unassigned early transfer counts", + ) + + +def test_gap_fill_validator_rejects_qrf_regime_evidence_tampering() -> None: + receipt, direction_name, key, _entity, _family, _targets = ( + _canonical_gap_fill_receipt_with_pattern_evidence() + ) + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="signed early QRF pattern evidence control", + ) + + forged = deepcopy(receipt) + forged["directions"][direction_name]["targets"][key]["qrf_pattern_evidence"][ + "patterns" + ][0]["target_regimes"][0]["regime"] = "negative_only" + with pytest.raises(ValueError, match="QRF pattern evidence SHA-256 mismatch"): + stacked_spine_module.validate_stacked_gap_fill_receipt( + forged, + boundary="tampered signed early QRF pattern evidence", + ) + + +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ("pattern_count", "evidence header is invalid"), + ("pattern_order", "name is not derived"), + ("recipient_rows", "recipient-row accounting is invalid"), + ("donor_rows", "metadata is invalid"), + ("weight_kind", "record binding is invalid"), + ("predictors", "outside canonical transfer authority"), + ("pattern_name", "name is not derived"), + ("model_target", "target order"), + ("record_family", "record binding is invalid"), + ("record_family_in_range", "record binding is invalid"), + ("record_family_out_of_range", "record binding is invalid"), + ("record_target", "record binding is invalid"), + ), +) +def test_gap_fill_validator_rejects_rehashed_qrf_pattern_structure_mutations( + mutation: str, + error_match: str, +) -> None: + receipt, direction_name, key, _entity, family, _targets = ( + _canonical_gap_fill_receipt_with_pattern_evidence() + ) + evidence = receipt["directions"][direction_name]["targets"][key][ + "qrf_pattern_evidence" + ] + patterns = evidence["patterns"] + if mutation == "pattern_count": + evidence["pattern_count"] += 1 + elif mutation == "pattern_order": + patterns.reverse() + elif mutation == "recipient_rows": + patterns[0]["recipient_rows"] += 1 + elif mutation == "donor_rows": + patterns[0]["donor_rows"] = 0 + elif mutation == "weight_kind": + evidence["record"]["weight_kind"] = "fabricated" + for pattern in patterns: + pattern["weight_kind"] = "fabricated" + elif mutation == "predictors": + patterns[0]["predictors"].append("fabricated_predictor") + elif mutation == "pattern_name": + patterns[0]["name"] = "pattern_00_00000000" + elif mutation == "model_target": + patterns[0]["target_regimes"][0]["model_target"] = "fabricated_target" + elif mutation == "record_family": + evidence["record"]["family"] = f"{family}__batch_forged" + elif mutation == "record_family_in_range": + evidence["record"]["family"] = f"{family}__batch_1" + elif mutation == "record_family_out_of_range": + evidence["record"]["family"] = f"{family}__batch_99" + else: + assert mutation == "record_target" + evidence["record"]["column"] = "fabricated_target" + unsigned = dict(evidence) + unsigned.pop("sha256") + evidence["sha256"] = stacked_spine_module._canonical_sha256(unsigned) + + with pytest.raises(ValueError, match=error_match): + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary=f"rehashed {mutation} QRF evidence", + ) + + +def test_receipt_only_qrf_validation_does_not_claim_seed_or_regime_replay() -> None: + receipt, direction_name, key, _entity, _family, _targets = ( + _canonical_gap_fill_receipt_with_pattern_evidence() + ) + evidence = receipt["directions"][direction_name]["targets"][key][ + "qrf_pattern_evidence" + ] + evidence["record"]["seed"] = 123 + for pattern in evidence["patterns"]: + pattern["seed"] += 123 + pattern["target_regimes"][0]["regime"] = "negative_only" + unsigned = dict(evidence) + unsigned.pop("sha256") + evidence["sha256"] = stacked_spine_module._canonical_sha256(unsigned) + + # Donor values and the top-level transfer seed are deliberately absent at + # this boundary. The receipt-only validator checks placement/vocabulary; + # the enclosing persisted manifest or late execution signature authenticates + # the reported values, as the late-signature mutation test below proves. + stacked_spine_module.validate_stacked_gap_fill_receipt( + receipt, + boundary="receipt-only reported seed and regime scope", + ) + + +@pytest.mark.parametrize( + ("mutation", "error_match"), + ( + ( + "stripped_direction_summary", + "stripped or misbound calibration summary evidence", + ), + ("stripped_target_evidence", "owner selection is misbound"), + ("deleted_target_receipt", "target surface is non-canonical"), + ), +) +def test_gap_fill_validator_rejects_stripped_calibration_evidence( + mutation: str, + error_match: str, +) -> None: + receipt = _canonical_gap_fill_calibration_receipt() + forged = deepcopy(receipt) + direction = next( + value + for value in forged["directions"].values() + if value["post_transfer_calibration"]["target_count"] > 0 + ) + if mutation == "stripped_direction_summary": + direction.pop("post_transfer_calibration") + else: + target_key = direction["post_transfer_calibration"]["targets"][0] + if mutation == "deleted_target_receipt": + direction["targets"].pop(target_key) + else: + direction["targets"][target_key].pop("post_transfer_calibration") + + with pytest.raises(ValueError, match=error_match): + stacked_spine_module.validate_stacked_gap_fill_receipt( + forged, + boundary=f"{mutation} regression", + ) + + def test_every_declared_direction_producer_precedes_its_activation_check() -> None: receipt = stacked_gap_fill_producer_schedule_receipt() assert receipt["status"] == "all_producers_precede_activation" @@ -2675,6 +3176,39 @@ def test_gap_fill_fills_both_directions_with_authority_receipts() -> None: assert unemployment["authorized_null_rows"] == int(acs_rows.sum()) assert unemployment["imputed_rows"] == int(acs_rows.sum()) assert unemployment["residual_null_rows"] == 0 + qrf_evidence = unemployment["qrf_pattern_evidence"] + assert qrf_evidence["pattern_count"] == len(qrf_evidence["patterns"]) + assert [pattern["name"] for pattern in qrf_evidence["patterns"]] == [ + f"pattern_{index:02d}_{pattern['name'].rsplit('_', 1)[1]}" + for index, pattern in enumerate(qrf_evidence["patterns"]) + ] + assert all( + pattern["target_regimes"] + == [ + { + "model_target": "unemployment_compensation", + "regime": "zero_inflated_positive", + } + ] + for pattern in qrf_evidence["patterns"] + ) + qrf_payload = dict(qrf_evidence) + assert qrf_payload.pop("sha256") == stacked_spine_module._canonical_sha256( + qrf_payload + ) + forged_unemployment = deepcopy(unemployment) + forged_unemployment["qrf_pattern_evidence"]["patterns"][0]["target_regimes"][0][ + "regime" + ] = "negative_only" + with pytest.raises(ValueError, match="QRF pattern evidence SHA-256 mismatch"): + stacked_spine_module._validate_acs_imputed_pattern_evidence( + forged_unemployment, + expected_entity="person", + expected_family="model_required_numeric", + expected_target="unemployment_compensation", + expected_family_targets=("unemployment_compensation",), + boundary="tampered ordinary early transfer receipt", + ) housing = directions["asec_housing_to_acs"] rent = housing["targets"]["person/housing/pre_subsidy_rent"] assert rent["imputed_rows"] == int((acs_rows & ~acs_gq_rows).sum()) @@ -4739,6 +5273,35 @@ def primary(frame: Frame): contracts=tuple(registry.values()), include_outputs=True, ) + completed_person = completed.table("person") + completed_person["unemployment_compensation"] = np.ones( + len(completed_person), + dtype=np.float64, + ) + completed_person["is_incapable_of_self_care"] = pd.Series( + True, + index=completed_person.index, + dtype="boolean", + ) + completed_person["tax_unit_role_input"] = pd.Series( + "DEPENDENT", + index=completed_person.index, + dtype="string", + ) + adult_recipient = completed_person[support_channel_column("person")].eq( + "acs" + ) & completed_person[support_clone_index_column("person")].eq(0) + completed_person.loc[ + adult_recipient, + "pre_subsidy_care_expenses", + ] = 0.0 + adult_carriers = ( + completed_person.loc[adult_recipient] + .groupby("person_tax_unit_id", sort=False, dropna=False) + .head(1) + .index + ) + completed_person.loc[adult_carriers, "pre_subsidy_care_expenses"] = 1.0 return stacked_spine_module.StackedPufPassResult( completed, { @@ -4809,26 +5372,194 @@ def transfer( derive_schedule_d=False, ) ) + required_predictors, _optional_predictors = ( + stacked_spine_module._acs_pattern_predictor_authority( + entity=group.entity, + family_targets=group.targets, + ) + ) + late_specs = { + spec.key: spec + for spec in post_transfer_calibration_runtime.POST_TRANSFER_CALIBRATION_SPECS.values() + if spec.stage == "late_transfer" + } + evidence_targets = tuple( + target + for target in group.targets + if f"{group.entity}/{group.family}/{target}" in late_specs + ) + model_targets = acs_transfer_module._model_target_names(evidence_targets) + pattern = AcsTransferPattern( + name="pattern_00_e3b0c442", + observed_optional_predictors=(), + predictors=required_predictors, + seed=0, + weight_kind="design", + donor_rows=1, + recipient_rows=1, + target_regimes=tuple((target, "positive_only") for target in model_targets), + ) + plain_pattern = replace(pattern, target_regimes=()) + synthetic_imputed_inputs = tuple( + AcsImputedInput( + column=target, + entity=group.entity, + family=group.family, + donor_spine="synthetic_late_executor_fixture", + donor_channel="asec", + predictors=pattern.predictors, + seed=pattern.seed, + weight_kind=pattern.weight_kind, + patterns=(pattern if target in evidence_targets else plain_pattern,), + imputed_recipient_rows=1, + ) + for target in group.targets + ) transfer_result = AcsTransferResult( frame=frame, - imputed_inputs=(), + imputed_inputs=synthetic_imputed_inputs, fit_records=(), deferred_inputs=(), resolved_donor_channel="asec", ) + policy_sha256 = post_transfer_calibration_runtime.post_transfer_calibration_policy_identity()[ + "sha256" + ] + target_receipts: dict[str, dict[str, object]] = {} + for target, record in zip( + group.targets, + synthetic_imputed_inputs, + strict=True, + ): + key = f"{group.entity}/{group.family}/{target}" + target_receipt: dict[str, object] = { + "authorized_null_rows": 1, + "imputed_rows": 1, + "unmodeled_rows": 0, + "residual_null_rows": 0, + } + if key in late_specs: + target_receipt["qrf_pattern_evidence"] = ( + stacked_spine_module._acs_imputed_pattern_evidence(record) + ) + target_receipts[key] = target_receipt + calibrated_keys = sorted(set(target_receipts) & set(late_specs)) + for key in calibrated_keys: + spec = late_specs[key] + constrained = spec.special_constraint != "none" + live_table = frame.table(spec.entity) + live_channel = live_table[support_channel_column(spec.entity)].astype(str) + live_clone = pd.to_numeric( + live_table[support_clone_index_column(spec.entity)], + errors="raise", + ) + live_reference = (live_channel.eq("asec") & live_clone.eq(0)).to_numpy( + dtype=bool + ) + live_recipient = (live_channel.eq("acs") & live_clone.eq(0)).to_numpy( + dtype=bool + ) + allowed_rows: np.ndarray | None = None + addition_rows: np.ndarray | None = None + if spec.special_constraint == ("adult_care_qualifying_one_per_tax_unit"): + mutable_series = pd.Series( + live_recipient, + index=live_table.index, + dtype=bool, + ) + allowed_series = ( + mutable_series + & acs_transfer_module.acs_adult_care_qualifying_rows(live_table) + ) + addition_series = ( + stacked_spine_module._one_candidate_per_adult_care_tax_unit( + frame, + mutable_rows=mutable_series, + allowed_rows=allowed_series, + ) + ) + allowed_rows = allowed_series.to_numpy(dtype=bool) + addition_rows = addition_series.to_numpy(dtype=bool) + elif spec.special_constraint == ( + "weeks_requires_positive_unemployment_compensation" + ): + allowed_rows = live_recipient & pd.to_numeric( + live_table["unemployment_compensation"], + errors="raise", + ).gt(0.0).to_numpy(dtype=bool) + addition_rows = allowed_rows.copy() + application = ( + post_transfer_calibration_runtime.apply_post_transfer_calibration( + frame, + entity=spec.entity, + family=spec.family, + target=spec.target, + reference_rows=live_reference, + recipient_rows=live_recipient, + mutable_rows=live_recipient, + allowed_carrier_rows=allowed_rows if constrained else None, + addition_candidate_rows=addition_rows if constrained else None, + ) + ) + frame = application.frame + calibration = application.receipt + scope = calibration["scope"] + constraint: dict[str, object] = {"constraint": spec.special_constraint} + if spec.special_constraint == ("adult_care_qualifying_one_per_tax_unit"): + constraint.update( + { + "qualifying_mutable_rows": scope["allowed_carrier_rows"], + "one_per_empty_tax_unit_addition_candidates": scope[ + "addition_candidate_rows" + ], + } + ) + elif spec.special_constraint == ( + "weeks_requires_positive_unemployment_compensation" + ): + constraint["positive_unemployment_mutable_rows"] = scope[ + "allowed_carrier_rows" + ] + owner: dict[str, object] = { + "stage": "late_transfer", + "reference_selection": "asec_origin_clone_0", + "recipient_selection": "acs_origin_clone_0", + "mutable_selection": "recipient_null_before_nonnull_after", + "reference_rows": scope["reference_rows"], + "recipient_rows": scope["recipient_rows"], + "mutable_rows": scope["mutable_rows"], + "constraint": constraint, + "context_binding": { + "scope": dict(scope), + "weights_sha256": calibration["weights"]["sha256"], + "live_output": ( + stacked_spine_module._post_transfer_selected_output_binding( + frame, + entity=spec.entity, + target=spec.target, + reference_rows=live_reference, + recipient_rows=live_recipient, + ) + ), + }, + "calibration": calibration, + } + if spec.special_constraint == ("adult_care_qualifying_one_per_tax_unit"): + owner["post_reconciliation"] = {"status": "verified_no_op"} + target_receipts[key]["post_transfer_calibration"] = owner return stacked_spine_module.StackedPostPufTransferResult( frame, { "producer": group.name, "ordered_targets": list(group.targets), - "targets": { - f"{group.entity}/{group.family}/{target}": { - "residual_null_rows": 0, - } - for target in group.targets + "targets": target_receipts, + "post_transfer_calibration": { + "policy_sha256": policy_sha256, + "target_count": len(calibrated_keys), + "targets": calibrated_keys, }, }, - transfer_result, + replace(transfer_result, frame=frame), ) monkeypatch.setattr( @@ -4928,6 +5659,204 @@ def test_real_late_executor_follows_canonical_order_and_finalizes_sources_once( ) +def test_late_executor_signature_rejects_qrf_regime_evidence_tampering( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + target_receipt = next( + target + for row in forged["execution"] + if row["kind"] == "late_transfer" + for target in row["producer_receipt"]["targets"].values() + if "qrf_pattern_evidence" in target + ) + target_receipt["qrf_pattern_evidence"]["patterns"][0]["target_regimes"][0][ + "regime" + ] = "negative_only" + + with pytest.raises(ValueError, match="callback-receipt SHA-256 mismatch"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="tampered signed QRF regime evidence", + ) + + +def test_post_puf_validator_rejects_unassigned_legacy_count_tampering( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + transfer = deepcopy(dict(result.receipt["post_puf_transfer"])) + target_key, target_receipt = next( + (key, target) + for group in transfer["groups"].values() + for key, target in group["targets"].items() + if "qrf_pattern_evidence" not in target + ) + target = target_key.rsplit("/", 1)[1] + aggregate_receipt = next( + receipt + for key, receipt in transfer["targets"].items() + if key.rsplit("/", 1)[1] == target + ) + for receipt in (target_receipt, aggregate_receipt): + receipt["imputed_rows"] = "forged" + + with pytest.raises(ValueError, match="ACS transfer row-count"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + transfer, + boundary="forged unassigned late transfer counts", + frame=result.frame, + ) + + +def test_late_executor_signature_rejects_generation_only_calibration_tampering( + monkeypatch: pytest.MonkeyPatch, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + forged = deepcopy(dict(result.receipt)) + owner = next( + target_receipt["post_transfer_calibration"] + for row in forged["execution"] + if row["kind"] == "late_transfer" + for target_receipt in row["producer_receipt"]["targets"].values() + if target_receipt.get("post_transfer_calibration") is not None + ) + calibration = owner["calibration"] + calibration["scope"]["input_values_sha256"] = "0" * 64 + unsigned = dict(calibration) + unsigned.pop("sha256") + calibration["sha256"] = stacked_spine_module._canonical_sha256(unsigned) + + with pytest.raises(ValueError, match="callback-receipt SHA-256 mismatch"): + stacked_spine_module.validate_stacked_late_producer_receipt( + forged, + boundary="tampered generation-only calibration evidence", + frame=result.frame, + expected_transition_authority_sha256=(result.transition_authority_sha256), + ) + + +@pytest.mark.parametrize("mutation", ("carrier", "amount")) +def test_late_transfer_rejects_rehashed_diagnostics_detached_from_live_output( + monkeypatch: pytest.MonkeyPatch, + mutation: str, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + transfer = deepcopy(dict(result.receipt["post_puf_transfer"])) + group_receipt = next( + group + for group in transfer["groups"].values() + if any(key.endswith("/disability_benefits") for key in group["targets"]) + ) + group_key = next( + key for key in group_receipt["targets"] if key.endswith("/disability_benefits") + ) + aggregate_key = next( + key for key in transfer["targets"] if key.endswith("/disability_benefits") + ) + target_receipts = ( + group_receipt["targets"][group_key], + transfer["targets"][aggregate_key], + ) + for target_receipt in target_receipts: + calibration = target_receipt["post_transfer_calibration"]["calibration"] + if mutation == "carrier": + carrier = calibration["carrier"] + recipient_total = calibration["weights"]["recipient_total"] + forged_mass = carrier["after_positive_mass"] / 2.0 + carrier["before_positive_mass"] = forged_mass + carrier["after_positive_mass"] = forged_mass + carrier["before_positive_share"] = forged_mass / recipient_total + carrier["after_positive_share"] = forged_mass / recipient_total + carrier["residual_after_minus_target"] = ( + forged_mass - carrier["target_positive_mass"] + ) + carrier["absolute_residual"] = abs(carrier["residual_after_minus_target"]) + else: + amount = calibration["amount"] + forged_quantiles = [ + value + 100.0 for value in amount["reference_quantiles"] + ] + amount["reference_quantiles"] = forged_quantiles + amount["recipient_before_quantiles"] = forged_quantiles.copy() + amount["recipient_after_quantiles"] = forged_quantiles.copy() + amount["qed_before"] = 0.0 + amount["qed_after"] = 0.0 + for anchor, value in zip( + amount["anchor_rows"], forged_quantiles, strict=True + ): + anchor["reference_value"] = value + unsigned = dict(calibration) + unsigned.pop("sha256") + calibration["sha256"] = stacked_spine_module._canonical_sha256(unsigned) + + with pytest.raises(ValueError, match="diagnostics do not match the live output"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + transfer, + boundary=f"rehashed {mutation} live-diagnostic forgery", + frame=result.frame, + ) + + +@pytest.mark.parametrize( + "mutation", + ("reference_scope", "output_values", "weights"), +) +def test_late_transfer_rejects_rehashed_context_detached_from_live_output( + monkeypatch: pytest.MonkeyPatch, + mutation: str, +) -> None: + result, _events, _finalizer_calls = _run_real_late_executor_fixture(monkeypatch) + transfer = deepcopy(dict(result.receipt["post_puf_transfer"])) + group_receipt = next( + group + for group in transfer["groups"].values() + if any(key.endswith("/disability_benefits") for key in group["targets"]) + ) + group_key = next( + key for key in group_receipt["targets"] if key.endswith("/disability_benefits") + ) + aggregate_key = next( + key for key in transfer["targets"] if key.endswith("/disability_benefits") + ) + owners = [ + group_receipt["targets"][group_key]["post_transfer_calibration"], + transfer["targets"][aggregate_key]["post_transfer_calibration"], + ] + seen: set[int] = set() + for owner in owners: + if id(owner) in seen: + continue + seen.add(id(owner)) + calibration = owner["calibration"] + context = owner["context_binding"] + if mutation == "reference_scope": + forged_count = owner["reference_rows"] + 1 + owner["reference_rows"] = forged_count + calibration["scope"]["reference_rows"] = forged_count + calibration["scope"]["reference_rows_sha256"] = "0" * 64 + context["scope"]["reference_rows"] = forged_count + context["scope"]["reference_rows_sha256"] = "0" * 64 + context["live_output"]["reference_rows"] = forged_count + elif mutation == "output_values": + calibration["scope"]["output_values_sha256"] = "0" * 64 + context["scope"]["output_values_sha256"] = "0" * 64 + else: + calibration["weights"]["sha256"] = "0" * 64 + context["weights_sha256"] = "0" * 64 + unsigned = dict(calibration) + unsigned.pop("sha256") + calibration["sha256"] = stacked_spine_module._canonical_sha256(unsigned) + + with pytest.raises(ValueError, match="match the live output"): + stacked_spine_module.validate_stacked_post_puf_transfer_receipt( + transfer, + boundary=f"rehashed {mutation} live-context forgery", + frame=result.frame, + ) + + def test_late_executor_authority_binds_every_transfer_bank_identity( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -5376,6 +6305,77 @@ def test_post_puf_transfer_preserves_complete_asec_source_producers() -> None: assert receipt["imputed_rows"] == int((~producer_rows).sum()) assert receipt["unmodeled_rows"] == 0 assert receipt["residual_null_rows"] == 0 + assert "qrf_pattern_evidence" not in receipt + record = next( + item + for item in result.transfer_result.imputed_inputs + if item.column == "is_pregnant" + ) + assert all(not pattern.target_regimes for pattern in record.patterns) + + +def test_late_calibration_owner_mutates_only_acs_clone_zero_transfer_cells() -> None: + frame = _post_puf_transfer_fixture() + person = frame.table("person") + channel = person[support_channel_column("person")].astype(str) + clone_index = pd.to_numeric( + person[support_clone_index_column("person")], + errors="raise", + ) + reference_rows = channel.eq("asec") & clone_index.eq(0) + recipient_rows = channel.eq("acs") & clone_index.eq(0) + target = "child_support_received" + + transferred_person = person.copy(deep=True) + transferred_person[target] = 0.0 + transferred_person.loc[reference_rows, target] = np.resize( + np.asarray([0.0, 100.0, 250.0]), + int(reference_rows.sum()), + ) + transferred_person.loc[recipient_rows, target] = np.resize( + np.asarray([0.0, 10.0, 20.0, 30.0]), + int(recipient_rows.sum()), + ) + before_person = transferred_person.copy(deep=True) + before_person.loc[recipient_rows, target] = np.nan + + def rebuild(person_table: pd.DataFrame) -> Frame: + tables = {entity: frame.table(entity) for entity in frame.entities} + tables["person"] = person_table + return Frame( + tables, + frame.schema, + {entity: frame.weights_for(entity) for entity in frame.weighted_entities}, + frame.strata, + mass_log=frame.mass_log, + metadata=frame.metadata, + ) + + transferred = rebuild(transferred_person) + calibrated, receipts = ( + stacked_spine_module._apply_stacked_post_transfer_calibrations( + rebuild(before_person), + transferred, + target_families={ + "person": { + "source_operator_child_support": (target,), + } + }, + stage="late_transfer", + ) + ) + calibrated_person = calibrated.table("person") + pd.testing.assert_series_equal( + calibrated_person.loc[~recipient_rows, target], + transferred_person.loc[~recipient_rows, target], + check_exact=True, + ) + receipt = receipts[f"person/source_operator_child_support/{target}"] + assert receipt["stage"] == "late_transfer" + assert receipt["reference_rows"] == int(reference_rows.sum()) + assert receipt["recipient_rows"] == int(recipient_rows.sum()) + assert receipt["mutable_rows"] == int(recipient_rows.sum()) + assert receipt["calibration"]["invariants"]["immutable_bytes_preserved"] @pytest.mark.parametrize( @@ -5571,6 +6571,102 @@ def test_gap_fill_banks_per_target_via_608_store(tmp_path) -> None: assert survey_receipt["targets"] +@pytest.mark.parametrize( + "use_target_bank", + [False, True], + ids=("ordinary", "banked"), +) +def test_gap_fill_scopes_qrf_evidence_off_wide_unassigned_family( + tmp_path: Path, + use_target_bank: bool, +) -> None: + stacked = _stacked_gap_fixture() + canonical_direction = next( + direction + for direction in stacked_spine_module.CANONICAL_STACKED_GAP_FILL_PLAN + if direction.name == "asec_survey_to_acs" + ) + puf_targets = canonical_direction.target_families["person"]["puf_tax_itemization"] + person = stacked.table("person").copy() + channel = person[support_channel_column("person")].astype(str) + donor_rows = channel.eq("asec") + for position, target in enumerate(puf_targets, start=1): + values = pd.Series(np.nan, index=person.index, dtype=np.float64) + values.loc[donor_rows] = np.arange(1, int(donor_rows.sum()) + 1) + position + person[target] = values + tables = {entity: stacked.table(entity) for entity in stacked.entities} + tables["person"] = person + frame = Frame( + tables, + stacked.schema, + {entity: stacked.weights_for(entity) for entity in stacked.weighted_entities}, + stacked.strata, + mass_log=stacked.mass_log, + metadata=stacked.metadata, + ) + direction = GapFillDirection( + name="asec_survey_to_acs", + recipient_channel="acs", + donor_channel="asec", + target_families={ + "person": { + "model_required_numeric": ("unemployment_compensation",), + "puf_tax_itemization": puf_targets, + } + }, + ) + target_banks = ( + { + direction.name: AcsTransferTargetBankStore( + tmp_path / "survey", + identity={"regression": "scoped-wide-gap-fill"}, + ) + } + if use_target_bank + else None + ) + + result = _gap_fill_with_test_authority( + frame, + plan=(direction,), + seed=578, + n_estimators=1, + target_banks=target_banks, + ) + + receipts = result.receipt["directions"][direction.name]["targets"] + taxable_key = "person/puf_tax_itemization/taxable_interest_income" + canonical_receipt = _canonical_gap_fill_calibration_receipt() + canonical_targets = canonical_receipt["directions"][direction.name]["targets"] + canonical_targets[taxable_key] = deepcopy(receipts[taxable_key]) + stacked_spine_module.validate_stacked_gap_fill_receipt( + canonical_receipt, + boundary=( + f"{'banked' if use_target_bank else 'ordinary'} generated " + "wide-family receipt" + ), + ) + + records = { + record.column: record + for record in result.transfer_results[direction.name].imputed_inputs + } + taxable = records["taxable_interest_income"] + unemployment = records["unemployment_compensation"] + assert taxable.family == "puf_tax_itemization__batch_1" + assert all(not pattern.target_regimes for pattern in taxable.patterns) + assert all( + tuple(target for target, _regime in pattern.target_regimes) + == ("unemployment_compensation",) + for pattern in unemployment.patterns + ) + assert "qrf_pattern_evidence" not in receipts[taxable_key] + assert ( + "qrf_pattern_evidence" + in receipts["person/model_required_numeric/unemployment_compensation"] + ) + + def test_clone_attachment_is_seeded_exact_and_pair_weighted() -> None: stacked = _stacked_gap_fixture() attached = clone_us_frame_for_puf_support( @@ -6961,7 +8057,7 @@ def test_self_digested_partial_authority_cannot_forge_production_identity() -> N GateReport((result,)).to_manifest() -@pytest.mark.parametrize("stale_version", (1, 2, 3, 4, 5, 6, 7, 8, 9)) +@pytest.mark.parametrize("stale_version", (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)) def test_self_consistent_stale_stacked_authority_versions_are_rejected( stale_version: int, ) -> None: @@ -6977,11 +8073,12 @@ def test_self_consistent_stale_stacked_authority_versions_are_rejected( metric_registry=canonical.metric_registry, joint_metric_registry=canonical.joint_metric_registry, support_profile=canonical.support_profile, + post_transfer_calibration=canonical.post_transfer_calibration, declared_form="CANONICAL", ) stale_receipt = stacked_spine_module._authority_receipt(stale) - assert stacked_spine_module.stacked_spine_authority_receipt()["version"] == 10 + assert stacked_spine_module.stacked_spine_authority_receipt()["version"] == 11 assert stale_receipt["version"] == stale_version assert stale_receipt["integrity_valid"] is True assert stale_receipt["digest_matches_declared"] is True @@ -7000,7 +8097,7 @@ def test_stacked_authority_binds_import_validated_late_producer_schedule() -> No receipt = stacked_spine_module.stacked_spine_authority_receipt() component = receipt["components"]["late_producer_schedule"] - assert receipt["version"] == 10 + assert receipt["version"] == 11 assert component["producer_count"] == 38 assert component["schedule_sha256"] == ( stacked_spine_module.CANONICAL_US_LATE_PRODUCER_SCHEDULE.sha256 @@ -7009,6 +8106,17 @@ def test_stacked_authority_binds_import_validated_late_producer_schedule() -> No assert component["digest_matches_declared"] is True +def test_stacked_authority_binds_post_transfer_calibration_policy() -> None: + receipt = stacked_spine_module.stacked_spine_authority_receipt() + component = receipt["components"]["post_transfer_calibration"] + + assert component["target_count"] == 9 + assert component["identity"] == ( + stacked_spine_module.post_transfer_calibration_policy_identity() + ) + assert component["digest_matches_declared"] is True + + def test_rebound_late_producer_schedule_invalidates_production_authority( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -7035,6 +8143,62 @@ def test_rebound_late_producer_schedule_invalidates_production_authority( ) +def test_rebound_post_transfer_calibration_invalidates_production_authority( + monkeypatch: pytest.MonkeyPatch, +) -> None: + live = dict(stacked_spine_module.post_transfer_calibration_policy_identity()) + live["tampered"] = True + monkeypatch.setattr( + stacked_spine_module, + "post_transfer_calibration_policy_identity", + lambda: live, + ) + + authority = stacked_spine_module._production_stacked_authority() + receipt = stacked_spine_module._authority_receipt(authority) + + assert receipt["canonical"] is False + assert ( + receipt["components"]["post_transfer_calibration"]["digest_matches_declared"] + is False + ) + with pytest.raises(ValueError, match="non-canonical stacked authority"): + stacked_spine_module._validate_production_authority_receipt( + receipt, + boundary="rebound post-transfer calibration", + ) + + +def test_rebound_live_calibration_registry_is_noncanonical_and_rejected( + monkeypatch: pytest.MonkeyPatch, +) -> None: + canonical_policy = ( + post_transfer_calibration_runtime.post_transfer_calibration_policy_identity() + ) + monkeypatch.setattr( + post_transfer_calibration_runtime, + "POST_TRANSFER_CALIBRATION_SPECS", + {}, + ) + + live_policy = ( + post_transfer_calibration_runtime.post_transfer_calibration_policy_identity() + ) + assert live_policy["targets"] == [] + assert live_policy["sha256"] != canonical_policy["sha256"] + + authority = stacked_spine_module._production_stacked_authority() + receipt = stacked_spine_module._authority_receipt(authority) + assert receipt["canonical"] is False + assert receipt["production_manifest_permitted"] is False + assert receipt["components"]["post_transfer_calibration"]["target_count"] == 0 + with pytest.raises(ValueError, match="non-canonical stacked authority"): + stacked_spine_module._validate_production_authority_receipt( + receipt, + boundary="rebound empty post-transfer calibration registry", + ) + + def test_rebound_anchor_aliases_cannot_replace_captured_canonical_authority( monkeypatch: pytest.MonkeyPatch, ) -> None: @@ -7433,9 +8597,7 @@ def test_stripped_noncanonical_receipt_cannot_escape_under_a_renamed_gate( GateReport((stripped,)).to_manifest() -def test_stripped_eight_component_authority_cannot_escape_under_a_renamed_gate() -> ( - None -): +def test_stripped_nine_component_authority_cannot_escape_under_a_renamed_gate() -> None: authority = stacked_spine_module.stacked_spine_authority_receipt() components = deepcopy(dict(authority["components"])) assert set(components) == { @@ -7447,6 +8609,7 @@ def test_stripped_eight_component_authority_cannot_escape_under_a_renamed_gate() "support_profile", "puf_capital_gains_tail_support_contract", "late_producer_schedule", + "post_transfer_calibration", } stripped = GateResult( name="renamed_stacked_battery", diff --git a/tools/audit_us_post_transfer_child_support_checkpoints.py b/tools/audit_us_post_transfer_child_support_checkpoints.py new file mode 100644 index 00000000..890d459c --- /dev/null +++ b/tools/audit_us_post_transfer_child_support_checkpoints.py @@ -0,0 +1,302 @@ +"""Audit checkpointed pkg3 child-support receipts without starting a build. + +The harness pins the assembled Frame and both child-support target checkpoints, +reconstructs only their native clone-0 post-transfer vectors, and runs the live +calibration kernel plus strict receipt validator. It performs no fit, DAG +execution, artifact write, or build. +""" + +from __future__ import annotations + +import argparse +import json +from dataclasses import dataclass +from pathlib import Path + +import numpy as np +import pandas as pd +from reproduce_us_post_transfer_weeks_checkpoint import ( + _ASSEMBLED_FILENAME, + _ASSEMBLED_SHA256, + _BANK_IDENTITY, + _FULL_POOL_CLONE_COUNT, + _load_target_draw, + _require_file_sha256, +) + +from microcosm.build.frame_checkpoint import load_frame_checkpoint +from microcosm.build.us_runtime.post_transfer_calibration import ( + calibrate_post_transfer_values, + post_transfer_calibration_spec_for_target, + validate_post_transfer_calibration_receipt, +) + +_EXPECTED_NATIVE_ROWS = 38_604 +_EXPECTED_REFERENCE_ROWS = 4_311 +_EXPECTED_RECIPIENT_ROWS = 34_293 +_EXPECTED_RECIPIENT_TOTAL = 79_926_522.10879111 +_EXPECTED_INVALID_MAXIMUM = 79_926_522.10879174 +_EXPECTED_MAXIMUM_RELATIONSHIP = "maximum_attainable_mass <= recipient_total" + + +@dataclass(frozen=True) +class _CheckpointCase: + target: str + source_column: str + relative_path: Path + file_sha256: str + identity_sha256: str + raw_sha256: str + + +_CASES = ( + _CheckpointCase( + target="child_support_expense", + source_column="CHSP_VAL", + relative_path=Path( + "late_producer_dag/person/source_operator_child_support/targets/" + "000__child_support_expense.h5" + ), + file_sha256=( + "d119075e19fb767f3d8d24c7c0149d0df1ed963774a4b93d96974a72b3ac9bfe" + ), + identity_sha256=( + "41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab" + ), + raw_sha256=("8b2845aff0aa0695d98ae30828523bf6bca9c5d4ed5d2d91d2d1a636bb917600"), + ), + _CheckpointCase( + target="child_support_received", + source_column="CSP_VAL", + relative_path=Path( + "late_producer_dag/person/source_operator_child_support/targets/" + "001__child_support_received.h5" + ), + file_sha256=( + "66120896d5793f3d737f9ffac2058e2196992e357f8d869f4b31b259d041b3aa" + ), + identity_sha256=( + "41e3a6e3877fda23107b27bcd85aa6dd95e0f341d1e4b079defa6847f90b4cab" + ), + raw_sha256=("ea7f2eebb430b654acc639ef6ee6ed482207ffd74d54ba3a47cb55056813a381"), + ), +) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--checkpoint-stage-root", + required=True, + type=Path, + help="Path to the SHA-addressed stacked stage containing the assembled H5.", + ) + parser.add_argument( + "--expect", + required=True, + choices=("invalid", "valid"), + help="Assert that both checkpointed child-support receipts have this state.", + ) + return parser.parse_args() + + +def _audit_case( + case: _CheckpointCase, + *, + bank_root: Path, + person: pd.DataFrame, + weights: np.ndarray, + entity_ids: np.ndarray, + reference: np.ndarray, + recipient: np.ndarray, +) -> dict[str, object]: + raw_draw = _load_target_draw( + bank_root / case.relative_path, + file_sha256=case.file_sha256, + identity_sha256=case.identity_sha256, + raw_sha256=case.raw_sha256, + ) + native_draw = raw_draw[: len(person)] + if ( + not np.isnan(native_draw[reference]).all() + or not np.isfinite(native_draw[recipient]).all() + ): + raise ValueError( + f"Pinned {case.target} draw no longer has null ASEC and finite ACS rows." + ) + direct = pd.to_numeric(person[case.source_column], errors="raise").to_numpy( + dtype=np.float64 + ) + if not np.isfinite(direct[reference]).all() or (direct[reference] < 0.0).any(): + raise ValueError(f"Pinned {case.source_column} reference values are invalid.") + values = np.where(reference, direct, native_draw).astype(np.float64) + mutable = recipient & np.isfinite(values) + if int(mutable.sum()) != _EXPECTED_RECIPIENT_ROWS: + raise ValueError(f"Pinned {case.target} mutable row count changed.") + + spec = post_transfer_calibration_spec_for_target( + entity="person", + target=case.target, + ) + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + ) + carrier = result.receipt["carrier"] + capacity = carrier["capacity"] + selection = carrier["selection"] + recipient_total = float(result.receipt["weights"]["recipient_total"]) + partition_mass = float( + capacity["fixed_positive_mass"] + + capacity["allowed_positive_mass_before"] + + capacity["addition_candidate_mass"] + ) + maximum = float(capacity["maximum_attainable_mass"]) + candidate_mass = float(selection["candidate_mass"]) + upper_mass = float(selection["upper_prefix_mass"]) + failed_relationships: list[str] = [] + if maximum > recipient_total: + failed_relationships.append("maximum_attainable_mass <= recipient_total") + if upper_mass > candidate_mass: + failed_relationships.append("upper_prefix_mass <= candidate_mass") + + validation_error: str | None = None + try: + validate_post_transfer_calibration_receipt( + result.receipt, + spec=spec, + boundary=f"Frame post-transfer calibration {spec.key}", + ) + except ValueError as error: + validation_error = str(error) + + return { + "artifact_sha256": { + "file": case.file_sha256, + "identity": case.identity_sha256, + "raw_draw": case.raw_sha256, + }, + "target": case.target, + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "mutable_rows": int(mutable.sum()), + "reference_positive_mass": carrier["reference_positive_mass"], + "target_positive_mass": carrier["target_positive_mass"], + "allowed_positive_rows": capacity["allowed_positive_rows_before"], + "allowed_positive_mass": capacity["allowed_positive_mass_before"], + "addition_candidate_rows": capacity["addition_candidate_rows"], + "addition_candidate_mass": capacity["addition_candidate_mass"], + "partition_endpoint_mass": partition_mass, + "maximum_attainable_mass": maximum, + "recipient_total": recipient_total, + "maximum_minus_recipient_total": maximum - recipient_total, + "partition_minus_maximum": partition_mass - maximum, + "selection_action": selection["action"], + "selected_rows": selection["selected_rows"], + "selected_mass": selection["selected_mass"], + "lower_prefix_mass": selection["lower_prefix_mass"], + "upper_prefix_mass": upper_mass, + "failed_relationships": failed_relationships, + "validation_error": validation_error, + "receipt_valid": validation_error is None, + } + + +def audit_checkpoints(checkpoint_stage_root: Path) -> dict[str, object]: + """Return strict receipt results for both pinned child-support targets.""" + + stage_root = checkpoint_stage_root.resolve() + assembled_path = stage_root / _ASSEMBLED_FILENAME + _require_file_sha256(assembled_path, _ASSEMBLED_SHA256) + frame = load_frame_checkpoint(assembled_path).frame + person = frame.table("person") + if len(person) != _EXPECTED_NATIVE_ROWS: + raise ValueError("Pinned assembled native person row count changed.") + + channels = person["person_support_channel"].astype(str) + clone_index = pd.to_numeric( + person["person_support_clone_index"], errors="raise" + ).to_numpy(dtype=np.int64) + reference = channels.eq("asec").to_numpy(dtype=bool) & (clone_index == 0) + recipient = channels.eq("acs").to_numpy(dtype=bool) & (clone_index == 0) + if ( + int(reference.sum()) != _EXPECTED_REFERENCE_ROWS + or int(recipient.sum()) != _EXPECTED_RECIPIENT_ROWS + or (reference & recipient).any() + or not np.all(reference | recipient) + ): + raise ValueError("Pinned assembled clone-0 support masks changed.") + + weights = ( + np.asarray(frame.resolve_weights("person").values, dtype=np.float64) + / _FULL_POOL_CLONE_COUNT + ) + entity_ids = person[frame.schema.entity_id_column("person")].to_numpy(copy=False) + bank_root = stage_root / "acs-transfer" / _BANK_IDENTITY + results = [ + _audit_case( + case, + bank_root=bank_root, + person=person, + weights=weights, + entity_ids=entity_ids, + reference=reference, + recipient=recipient, + ) + for case in _CASES + ] + return { + "assembled_sha256": _ASSEMBLED_SHA256, + "targets": results, + "all_receipts_valid": all(result["receipt_valid"] for result in results), + } + + +def main() -> int: + args = _parse_args() + audit = audit_checkpoints(args.checkpoint_stage_root) + print(json.dumps(audit, indent=2, sort_keys=True, allow_nan=False)) + targets = audit["targets"] + expected_valid = args.expect == "valid" + states_match = all(result["receipt_valid"] is expected_valid for result in targets) + if not states_match: + raise SystemExit( + f"Expected each child-support receipt to be {args.expect}, but got " + f"{[result['receipt_valid'] for result in targets]}." + ) + if expected_valid: + proof_matches = all( + not result["failed_relationships"] + and result["validation_error"] is None + and result["recipient_total"] == _EXPECTED_RECIPIENT_TOTAL + and result["maximum_attainable_mass"] == _EXPECTED_RECIPIENT_TOTAL + and result["partition_endpoint_mass"] == _EXPECTED_INVALID_MAXIMUM + for result in targets + ) + else: + proof_matches = all( + result["failed_relationships"] == [_EXPECTED_MAXIMUM_RELATIONSHIP] + and result["validation_error"] + == ( + "Frame post-transfer calibration " + f"person/source_operator_child_support/{result['target']}: " + "match-reference carrier capacity relationships are invalid." + ) + and result["recipient_total"] == _EXPECTED_RECIPIENT_TOTAL + and result["maximum_attainable_mass"] == _EXPECTED_INVALID_MAXIMUM + for result in targets + ) + if not proof_matches: + raise SystemExit( + f"The {args.expect} receipts no longer match the pinned relationship proof." + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/build_us_multispine_pool.py b/tools/build_us_multispine_pool.py index 9508e588..4cd14c64 100644 --- a/tools/build_us_multispine_pool.py +++ b/tools/build_us_multispine_pool.py @@ -3827,7 +3827,13 @@ def _json_ready(value: object) -> object: if isinstance(value, Mapping): if any(not isinstance(key, str) for key in value): raise ValueError("Build receipt mappings must use string JSON keys.") - return {key: _json_ready(item) for key, item in value.items()} + return { + key: _json_ready(item) + for key, item in value.items() + # The opt-in stacked audit field did not exist on legacy transfer + # provenance. Preserve the default build-receipt JSON schema. + if key != "target_regimes" or item + } if is_dataclass(value) and not isinstance(value, type): # Walk fields directly rather than through dataclasses.asdict(), whose # recursive deepcopy cannot serialize immutable MappingProxyType @@ -3835,6 +3841,9 @@ def _json_ready(value: object) -> object: return { field.name: _json_ready(getattr(value, field.name)) for field in fields(value) + # Match the mapping path above while avoiding dataclasses.asdict() + # for immutable authority records. + if field.name != "target_regimes" or getattr(value, field.name) } if isinstance(value, (list, tuple)): return [_json_ready(item) for item in value] diff --git a/tools/reproduce_us_post_transfer_weeks_checkpoint.py b/tools/reproduce_us_post_transfer_weeks_checkpoint.py new file mode 100644 index 00000000..42c22a8b --- /dev/null +++ b/tools/reproduce_us_post_transfer_weeks_checkpoint.py @@ -0,0 +1,286 @@ +"""Replay the pkg3 weeks receipt from checkpoints without starting a build. + +The harness is intentionally pinned to the preserved 1% failure artifacts. It +loads the assembled frame and two identity-bound ACS target checkpoints, then +reconstructs only the clone-0 vectors consumed by the post-transfer kernel. +It never fits a model, executes the late-producer DAG, or writes an artifact. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +from pathlib import Path + +import h5py +import numpy as np +import pandas as pd + +from microcosm.build.frame_checkpoint import load_frame_checkpoint +from microcosm.build.us_runtime.post_transfer_calibration import ( + calibrate_post_transfer_values, + post_transfer_calibration_spec_for_target, + validate_post_transfer_calibration_receipt, +) + +_ASSEMBLED_FILENAME = "assembled.checkpoint.h5" +_ASSEMBLED_SHA256 = "5ce1815fc44dc43c7c24ccf27526852b8f1bddbdfe371255410a22f9b56ac015" +_BANK_IDENTITY = "65c58b1c3fb282cbf7a814dad32481c1fe2de977ad747fe6576bdfc40b712019" +_UC_RELATIVE_PATH = Path("asec_survey_to_acs/targets/019__unemployment_compensation.h5") +_UC_FILE_SHA256 = "dc6637936ed4bd0322d38eaa3a4920fd137565f314387db3b3fdc7dfd6bc3086" +_UC_IDENTITY_SHA256 = "708722093ca610426175998d50bbb6663585b07ffef912899f17adc90520f51f" +_UC_RAW_SHA256 = "e32d1559668e10b24abad8e1d639e4dbade964a712925bfe8f56d3136b839840" +_WEEKS_RELATIVE_PATH = Path( + "late_producer_dag/person/source_operator_weeks_unemployed/targets/" + "000__weeks_unemployed.h5" +) +_WEEKS_FILE_SHA256 = "898397733aa3e5d8ec7d6679cb16a0504e826e25d23ca2c788f4397e0e061a43" +_WEEKS_IDENTITY_SHA256 = ( + "d0d554ba05045e39a07f0f9515c83bbf754f067df12b8247f4bf3866162c4bdd" +) +_WEEKS_RAW_SHA256 = "0214c8dcbc118676336069b906a07ee6145f2178542b6c5b4fb5899ad62d09f3" +_EXPECTED_NATIVE_ROWS = 38_604 +_EXPECTED_REFERENCE_ROWS = 4_311 +_EXPECTED_RECIPIENT_ROWS = 34_293 +_EXPECTED_ALLOWED_ROWS = 32 +_EXPECTED_REFERENCE_POSITIVE_ROWS = 134 +_EXPECTED_RECIPIENT_POSITIVE_ROWS = 24 +_FULL_POOL_CLONE_COUNT = 2 +_EXPECTED_ERROR = ( + "Frame post-transfer calibration " + "person/source_operator_weeks_unemployed/weeks_unemployed: " + "match-reference carrier capacity relationships are invalid." +) +_EXPECTED_INVALID_CANDIDATE_MASS = 85_676.23791782455 +_EXPECTED_PREFIX_MASS = 85_676.23791782456 +_EXPECTED_VALID_MAXIMUM_MASS = 85_676.23791782453 + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--checkpoint-stage-root", + required=True, + type=Path, + help="Path to the SHA-addressed stacked stage containing the assembled H5.", + ) + parser.add_argument( + "--expect", + required=True, + choices=("invalid", "valid"), + help="Assert the current kernel either reproduces or fixes the receipt.", + ) + return parser.parse_args() + + +def _file_sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for block in iter(lambda: stream.read(1024 * 1024), b""): + digest.update(block) + return digest.hexdigest() + + +def _require_file_sha256(path: Path, expected: str) -> None: + observed = _file_sha256(path) + if observed != expected: + raise ValueError( + f"Checkpoint SHA-256 changed for {path}: {observed} != {expected}." + ) + + +def _load_target_draw( + path: Path, + *, + file_sha256: str, + identity_sha256: str, + raw_sha256: str, +) -> np.ndarray: + _require_file_sha256(path, file_sha256) + with h5py.File(path, mode="r") as h5: + metadata = json.loads(bytes(h5["metadata_json"][...]).decode("utf-8")) + raw_bits = np.asarray(h5["raw_draw_bits"], dtype=" dict[str, object]: + """Return the exact capacity evidence and validator outcome for the replay.""" + + stage_root = checkpoint_stage_root.resolve() + assembled_path = stage_root / _ASSEMBLED_FILENAME + _require_file_sha256(assembled_path, _ASSEMBLED_SHA256) + frame = load_frame_checkpoint(assembled_path).frame + person = frame.table("person") + if len(person) != _EXPECTED_NATIVE_ROWS: + raise ValueError( + f"Assembled person rows changed: {len(person)} != {_EXPECTED_NATIVE_ROWS}." + ) + + bank_root = stage_root / "acs-transfer" / _BANK_IDENTITY + unemployment = _load_target_draw( + bank_root / _UC_RELATIVE_PATH, + file_sha256=_UC_FILE_SHA256, + identity_sha256=_UC_IDENTITY_SHA256, + raw_sha256=_UC_RAW_SHA256, + ) + weeks_full = _load_target_draw( + bank_root / _WEEKS_RELATIVE_PATH, + file_sha256=_WEEKS_FILE_SHA256, + identity_sha256=_WEEKS_IDENTITY_SHA256, + raw_sha256=_WEEKS_RAW_SHA256, + ) + if len(unemployment) != len(person) or len(weeks_full) < len(person): + raise ValueError("Target checkpoints do not cover the assembled native rows.") + + channels = person["person_support_channel"].astype(str) + clone_index = pd.to_numeric( + person["person_support_clone_index"], errors="raise" + ).to_numpy(dtype=np.int64) + reference = channels.eq("asec").to_numpy(dtype=bool) & (clone_index == 0) + recipient = channels.eq("acs").to_numpy(dtype=bool) & (clone_index == 0) + if (reference & recipient).any() or not np.all(reference | recipient): + raise ValueError("Assembled clone-0 ASEC/ACS masks are not exhaustive.") + + direct_weeks = pd.to_numeric(person["LKWEEKS"], errors="raise").to_numpy( + dtype=np.float64 + ) + direct_weeks = np.where(direct_weeks == -1.0, 0.0, direct_weeks) + values = np.where(reference, direct_weeks, weeks_full[: len(person)]).astype( + np.float64 + ) + mutable = recipient & np.isfinite(values) + allowed = mutable & (unemployment > 0.0) + weights = ( + np.asarray(frame.resolve_weights("person").values, dtype=np.float64) + / _FULL_POOL_CLONE_COUNT + ) + entity_ids = person[frame.schema.entity_id_column("person")].to_numpy(copy=False) + + observed_counts = { + "reference_rows": int(reference.sum()), + "recipient_rows": int(recipient.sum()), + "mutable_rows": int(mutable.sum()), + "allowed_addition_rows": int(allowed.sum()), + "reference_positive_rows": int((reference & (values > 0.0)).sum()), + "recipient_positive_rows_before": int((recipient & (values > 0.0)).sum()), + "disallowed_positive_rows_before": int( + (recipient & (values > 0.0) & ~allowed).sum() + ), + } + expected_counts = { + "reference_rows": _EXPECTED_REFERENCE_ROWS, + "recipient_rows": _EXPECTED_RECIPIENT_ROWS, + "mutable_rows": _EXPECTED_RECIPIENT_ROWS, + "allowed_addition_rows": _EXPECTED_ALLOWED_ROWS, + "reference_positive_rows": _EXPECTED_REFERENCE_POSITIVE_ROWS, + "recipient_positive_rows_before": _EXPECTED_RECIPIENT_POSITIVE_ROWS, + "disallowed_positive_rows_before": _EXPECTED_RECIPIENT_POSITIVE_ROWS, + } + if observed_counts != expected_counts: + raise ValueError( + f"Pinned replay row counts changed: {observed_counts} != {expected_counts}." + ) + + spec = post_transfer_calibration_spec_for_target( + entity="person", target="weeks_unemployed" + ) + result = calibrate_post_transfer_values( + values, + weights, + entity_ids, + spec=spec, + reference_rows=reference, + recipient_rows=recipient, + mutable_rows=mutable, + allowed_carrier_rows=allowed, + addition_candidate_rows=allowed, + ) + carrier = result.receipt["carrier"] + capacity = carrier["capacity"] + selection = carrier["selection"] + candidate_mass = float(capacity["addition_candidate_mass"]) + upper_mass = float(selection["upper_prefix_mass"]) + failed_relationships = [] + if not 0.0 <= float(selection["lower_prefix_mass"]) <= upper_mass: + failed_relationships.append("0 <= lower_prefix_mass <= upper_prefix_mass") + if not upper_mass <= candidate_mass: + failed_relationships.append("upper_prefix_mass <= addition_candidate_mass") + + validation_error: str | None = None + try: + validate_post_transfer_calibration_receipt( + result.receipt, + spec=spec, + boundary=f"Frame post-transfer calibration {spec.key}", + ) + except ValueError as error: + validation_error = str(error) + if validation_error != _EXPECTED_ERROR: + raise + + return { + "artifact_sha256": { + "assembled": _ASSEMBLED_SHA256, + "unemployment_compensation": _UC_FILE_SHA256, + "weeks_unemployed": _WEEKS_FILE_SHA256, + }, + "counts": observed_counts, + "weights": result.receipt["weights"], + "reference_positive_mass": carrier["reference_positive_mass"], + "target_positive_mass": carrier["target_positive_mass"], + "before_positive_mass": carrier["before_positive_mass"], + "after_positive_mass": carrier["after_positive_mass"], + "addition_candidate_mass": candidate_mass, + "maximum_attainable_mass": capacity["maximum_attainable_mass"], + "selected_prefix_mass": selection["selected_mass"], + "lower_prefix_mass": selection["lower_prefix_mass"], + "upper_prefix_mass": upper_mass, + "upper_minus_candidate_mass": upper_mass - candidate_mass, + "failed_relationships": failed_relationships, + "validation_error": validation_error, + "receipt_valid": validation_error is None, + } + + +def main() -> int: + args = _parse_args() + replay = replay_checkpoint(args.checkpoint_stage_root) + print(json.dumps(replay, indent=2, sort_keys=True, allow_nan=False)) + observed = "valid" if replay["receipt_valid"] else "invalid" + if observed != args.expect: + raise SystemExit( + f"Expected {args.expect} receipt, but replay produced {observed}." + ) + if args.expect == "invalid" and ( + replay["failed_relationships"] + != ["upper_prefix_mass <= addition_candidate_mass"] + or replay["validation_error"] != _EXPECTED_ERROR + or replay["addition_candidate_mass"] != _EXPECTED_INVALID_CANDIDATE_MASS + or replay["upper_prefix_mass"] != _EXPECTED_PREFIX_MASS + ): + raise SystemExit("Invalid replay no longer matches the pinned weeks failure.") + if args.expect == "valid" and ( + replay["failed_relationships"] + or replay["validation_error"] is not None + or replay["addition_candidate_mass"] != _EXPECTED_PREFIX_MASS + or replay["upper_prefix_mass"] != _EXPECTED_PREFIX_MASS + or replay["maximum_attainable_mass"] != _EXPECTED_VALID_MAXIMUM_MASS + ): + raise SystemExit("Valid replay no longer matches the pinned weeks repair.") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/spec_engine_coverage.py b/tools/spec_engine_coverage.py index d4185ff5..c06aee91 100644 --- a/tools/spec_engine_coverage.py +++ b/tools/spec_engine_coverage.py @@ -41,7 +41,7 @@ REPORT_SCHEMA_VERSION = 3 EXPECTED_POINTER_INVENTORY_SHA256 = ( - "b9d2932e0ae5efb6637be7b5d90e33b2e159ffb807ee50ff36ee2b51296cfdc4" + "2daa3ee07ac2e5d5ab731348edbca8c7a58438e9819d1ac3707070c7500a1c63" ) DEFAULT_REPORT_PATH = ( Path(__file__).resolve().parents[1] diff --git a/tools/us_bundle_generation/imputation.py b/tools/us_bundle_generation/imputation.py index b6ce35d7..945f97af 100644 --- a/tools/us_bundle_generation/imputation.py +++ b/tools/us_bundle_generation/imputation.py @@ -266,6 +266,125 @@ def _canonical_sha256(value: object) -> str: ).hexdigest() +def _post_draw_calibration_policy() -> dict[str, object]: + """Author the constants-era post-transfer policy without importing it.""" + + payload: dict[str, object] = { + "artifact_kind": "microcosm_us_post_transfer_calibration_policy", + "schema_version": 1, + "scope": { + "reference": "asec_origin_clone_0", + "recipient": "acs_origin_clone_0", + "mutable": "caller_supplied_target_cells", + "provenance_masks": "caller_supplied_no_internal_inference", + "constraint_masks": "caller_supplied_hash_bound", + "value_dtype": "float64_byte_contract", + "zero_weight_rows": "byte_exact", + }, + "quantiles": [0.10, 0.25, 0.50, 0.75, 0.90], + "carrier_selection": { + "match_reference": "weighted_positive_prevalence_nearest_prefix", + "removal_order": "positive_amount_descending_then_entity_id", + "addition_order": "entity_id", + "equal_distance": "lower_mass", + }, + "amount_mapping": { + "leg": "positive", + "recipient_rank": "weighted_full_recipient_positive_upper_cdf", + "inverse_cdf": "left", + "exact_quantile_anchors": [0.10, 0.25, 0.50, 0.75, 0.90], + "infeasible_anchor_handling": "frame_owner_fail_closed", + "output_support": "reference_positive_values_only", + }, + "targets": [ + { + "entity": "person", + "family": "adult_care", + "target": "pre_subsidy_care_expenses", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": ("adult_care_qualifying_one_per_tax_unit"), + }, + { + "entity": "person", + "family": "model_required_numeric", + "target": "unemployment_compensation", + "stage": "early_gap_fill", + "carrier_mode": "preserve_recipient", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "person", + "family": "source_operator_child_support", + "target": "child_support_expense", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "person", + "family": "source_operator_child_support", + "target": "child_support_received", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "person", + "family": "source_operator_disability_benefits", + "target": "disability_benefits", + "stage": "late_transfer", + "carrier_mode": "preserve_recipient", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "person", + "family": "source_operator_prior_year_income", + "target": "self_employment_income_last_year", + "stage": "early_gap_fill", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "person", + "family": "source_operator_weeks_unemployed", + "target": "weeks_unemployed", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": ( + "weeks_requires_positive_unemployment_compensation" + ), + }, + { + "entity": "person", + "family": "source_operator_workers_compensation", + "target": "workers_compensation", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + { + "entity": "spm_unit", + "family": "source_operator_energy_subsidy", + "target": "spm_unit_energy_subsidy", + "stage": "late_transfer", + "carrier_mode": "match_reference", + "negative_leg": "byte_exact", + "special_constraint": "none", + }, + ], + } + return {**payload, "sha256": _canonical_sha256(payload)} + + def _without_sha256(value: Mapping[str, object]) -> dict[str, object]: """Copy a canonical receipt body without its derived digest snapshot.""" @@ -1000,8 +1119,7 @@ def _normalise_virtual_resource( clone_attachment.get("fraction") != _LEGACY_CLONE_ATTACHMENT_FRACTION or clone_attachment.get("seed") != _LEGACY_CLONE_ATTACHMENT_SEED or clone_attachment.get("puf_clone_index") != 1 - or clone_attachment.get("support_channels") - != ["asec", "puf_tax_detail"] + or clone_attachment.get("support_channels") != ["asec", "puf_tax_detail"] ): raise RuntimeError( "Primary clone attachment differs from the spine-owned contract." @@ -1240,8 +1358,7 @@ def _assert_family_node_kinds( ) if unlinked: raise RuntimeError( - "Modeled producer nodes must resolve exactly one family: " - f"{unlinked!r}." + f"Modeled producer nodes must resolve exactly one family: {unlinked!r}." ) @@ -1307,9 +1424,7 @@ def _compile_node_outputs( raise RuntimeError("Producer graph must be the document's authored graph.") return { producer: [deepcopy(dict(row)) for row in rows] - for producer, rows in compile_producer_outputs( - {"imputation": document} - ).items() + for producer, rows in compile_producer_outputs({"imputation": document}).items() } @@ -1733,9 +1848,7 @@ def _spine_support_role( if _mapping_like(value, "spine support role").get("id") == role_id ] if len(matches) != 1: - raise RuntimeError( - f"Spine must declare exactly one support role {role_id!r}." - ) + raise RuntimeError(f"Spine must declare exactly one support role {role_id!r}.") return matches[0] @@ -1906,6 +2019,7 @@ def derive_primary_effective_predictor_tuples( return _derive_primary_effective_predictor_tuples(document) + def project_imputation_legacy_payloads( document: Mapping[str, object], *, @@ -1922,6 +2036,7 @@ def project_imputation_legacy_payloads( bundle_document=bundle_document, ) + def _split_after() -> list[dict[str, object]]: late_batches = [ group @@ -2102,15 +2217,13 @@ def _assert_invariants( spine_document, role_id=_PUF_ATTACHMENT_REF["support_role"], ) - attachment = _mapping_like( - attachment_role["attachment"], "spine PUF attachment" - ) + attachment = _mapping_like(attachment_role["attachment"], "spine PUF attachment") attachment_fraction = _mapping_like( attachment["fraction"], "spine PUF attachment fraction" )["default"] - attachment_seed = _mapping_like( - attachment["seed"], "spine PUF attachment seed" - )["default"] + attachment_seed = _mapping_like(attachment["seed"], "spine PUF attachment seed")[ + "default" + ] build_model_seed = _build_model_seed_default( _mapping_like( _mapping_like( @@ -2221,6 +2334,7 @@ def build_imputation() -> dict[str, object]: "max_samples_leaf": None, "zero_atol": DEFAULT_ZERO_ATOL, }, + "post_draw_calibration": _post_draw_calibration_policy(), } }, "chaining": {