Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686) - #747
Draft
juaristi22 wants to merge 15 commits into
Draft
Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686)#747juaristi22 wants to merge 15 commits into
juaristi22 wants to merge 15 commits into
Conversation
The frozen instruments were pinned at the incumbent data package's 1.56.14, which carries uk-data#461: from the 2024-25 FRS release the raw benunit table is no longer ordered by sernum, so every benunit-level variable — benunit_id included — landed on the wrong benefit unit relative to the model's sorted-id entity order. The parity screen compares unweighted nonzero shares, which are invariant under a row permutation, so it cannot see this. is_married is one of the 145 columns it compares and reads a byte-identical 0.256587 on both artifacts while every value sits on a different row. Signing whole-spine parity against 1.56.14 would have frozen the upstream defect into the contract as though it were correct. Verified on the new artifact before trusting it: entity counts and column surface unchanged (113,617 / 61,223 / 52,846; 145 layers), clone_index uniformly 0 so it is still the pre-clone artifact that compares row-for-row with the spine grain, and benunit_id now sorted ascending where 1.56.14 was not. The identity is corroborated three ways — the HF LFS metadata at the tagged revision, the repo's own releases/1.56.16/release_manifest.json, and a local hash. Reference-side movement is small: 39 of 145 shares move, none by more than 0.0046, so the +/-0.02 screen is undisturbed. The licensed weighted register moves on all 128 comparable columns, but that is the already-signed register-realization class — 1.56.15 changed the UC caseload targets and the incumbent's calibration re-solves with unseeded dropout — and stays far inside the gross-mass fence. No gate threshold moves here; #686 arms the re-measured baselines later. uk/frs_release.json is deliberately untouched: its revision names the raw UKDS zip, a different artifact that did not change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
FRS 2024-25 retired CWATAMT and CSEWAMT ("Final value after discount").
The headers survive but carry no data in any of the 16,288 households,
and the FRS replaced them with the Scotland-only CWATAMT1/CSEWAMT1
("Weeklyised gross annual dom. water/sew. charge on bill", "DV created
in 2024-25 as variable was removed from the dataset").
Two things were live. The incumbent adds the retired cell before
filling, so a wholly blank CSEWAMT propagates NaN and zeroes the charge
for all 1,663 Scottish households that have one; our per-column fill
left it standing, which is the whole +0.1009 share divergence flagged
on #736 — we are correct, the incumbent is defective, and the gap
reproduces on the raw tab at +0.1021 before composition. Separately our
own level was short: CWATAMTD is the water charge alone, about GBP 185
per Scottish household against roughly GBP 490 for England and Wales.
Both consumers now call one helper, so the amount netted from the
council tax bill is exactly the amount charged as water and sewerage —
an inconsistency the incumbent still has, since its netting fills per
column while its charge fills after the addition. The helper adds
CSEWAMT1 discounted at the household's own observed CWATAMTD/CWATAMT1,
which keeps the retired cells' after-discount meaning instead of
silently switching to a gross basis. The factor is well behaved: range
(1/3, 1], never above 1, for the 1,641 households with a positive gross
bill. The other two domains cannot be moved by it — the 22 with a
recorded CWATAMTD but no gross cell have zero sewerage, and the 21 with
no council-tax cells stay at zero.
The level fix does not move the nonzero share (0.8783767190569745
either way, matching the existing spine evidence to every digit), so
the share difference stands alone as an incumbent-defect divergence.
The fixtures supplied a non-missing CSEWAMT and so never exercised what
the tab contains; they now carry the retired cells blank, and a
regression test pins that a blank retired cell and an absent one give
the same non-zero answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Disclosure-controlled evidence for the two changes above: the pin identity and its three-way corroboration, the structural verification of the new artifact, the defect footprint across benunit/person/ household, the reference-side share movement, the licensed weighted register's digests and relative drift, and the FRS 2024-25 water cell retirement with the factor domain and level comparison. Digests, column counts, unweighted shares and relative deltas only per CD171 5.2.1; the licensed register itself stays uncommitted under the UKDS EUL. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The whole-spine comparison has one rule: anything differing between the spine and the frozen incumbent that is not signed is a defect. That rule needs somewhere to read the signatures from, so this adds the register, its loader, and the validation that keeps it trustworthy. Each entry names its class, the exact surface and columns the difference is expected to appear on, disclosure-safe magnitude evidence, the adjudicator and the date. The loader enforces the three vocabularies, unique kebab-case ids and ISO dates, and a test refuses a column-surface entry that names no columns — an unscoped entry would quietly absorb unrelated divergences, which is precisely the failure this register exists to prevent. It sits above the per-gate reviewed-exclusion registers rather than replacing them. Those are per-gate, per-reference and expiring, because a suppression must not outlive its reason; a signed difference is a permanent adjudicated fact. So expires_on is rejected outright, with the error pointing at the exclusion registers instead. Seeded with the two Scottish water adjudications, deliberately scoped apart: the incumbent's NaN-zeroing signs the share surface, the successor-cell level change signs weighted totals and leaves the share — which it does not move — unsigned. The E4-E8 method classes and the #723 beyond-band columns are transcribed as each is re-measured against the re-pinned reference and scoped to what is actually observed, rather than carried across on prose classification. The country package declares everything it ships, so the resource is registered there; that moves the UK spec bundle sha, re-pinned in the same change. Wheel-checked: the resource ships beside its siblings. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The swap decision rests on this comparison, so both halves of it live here: a candidate mode on the existing extractor, and the diff that holds its output to the signed register. The candidate mode reuses build_reference and build_weighted_totals unchanged and only swaps the identity block. That is the point — if the two sides were measured by different producers, the diff would be between two measurement methods rather than two artifacts. A candidate is identified by its own sha256 instead of being checked against the incumbent pin, and the mode cannot write the committed reference: it refuses any destination inside the country package, and --check is refused outright because a candidate can never satisfy a check against the incumbent pin. verify_uk_spine_parity.py compares the record-count identity exactly, per-column nonzero shares at the reference's own six-decimal grain with the column-set difference both ways, and optionally the two licensed weighted registers as relative deltas only. Every difference must match a register entry; anything else is a defect and exits 1. Two fences stop the verdict being manufactured. The reference side is always the committed instrument, and a candidate extraction claiming the incumbent's own sha256 is refused rather than compared — a copied reference would pass by construction. --strict, the swap-acceptance posture, also fails when a register entry matched nothing, so the register cannot decay into a blanket amnesty as the spine changes. Verified end to end against the E8 spine artifact: 142 columns compared, the household record-count identity holding while the known donor- composition deltas show on persons and benefit units, and water_and_sewerage_charges reproducing at +0.100897 and binding to its signed entry rather than counting as unsigned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The swap comparison asks a different question from payload identity: the control and candidate builds should share a surface exactly and differ in values only where a difference has been adjudicated. This re-verdicts the same measurements rather than relaxing them. Every structural predicate stays strict — same keys and stored kinds, row counts, column lists in order, dtypes, indexes, root-attribute names — and each differing column or root attribute must name an entry in the committed signed-differences register. A signature excuses a differing value, never a differing surface, which is pinned by a test that supplies a signature for an added column and still expects failure. payload_identical is still computed and reported in both modes, so a structure-only receipt stays comparable with a full-mode one, and --signed-differences is refused outside the mode rather than silently ignored. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Not an acceptance run — the E8 spine predates both the re-pin and the water fix, and the register is deliberately seeded with only the two adjudications made so far. It records that the instrument behaves as specified on real artifacts before the licensed ladder depends on it: the household record-count identity holds, the known donor-composition deltas surface on persons and benefit units, the E9 derived-benefit columns are still the three missing on the candidate side, the water column binds to its signature instead of counting as unsigned, and the weighted-totals entry correctly reports as unused when no weighted registers were supplied. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both UK resource tuples in test_country_spec.py enumerate the package's resources exactly, so #686's signed-differences register has to appear in them. Caught by the full build-shard run rather than the targeted ones. The first test's name records the #717 question it was written to answer, but what it does now is pin the whole legacy-JSON list; a note says so, so the next reader is not puzzled by a resource arriving in a test that says nothing was added. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
L0 ladder built from raw licensed tabs on this branch: smoke, dev and full all clean, every attempt landing a Logbook row. The record-count identity closes exactly at full scale — (16,288 + 10,000) x 2 + 270 = 52,846 — and the Scottish water fix reproduces at every rung. Parity against the re-pinned reference finds 26 columns beyond the band, against 27 in the #723 screen, which is what a re-pin that moved no reference share by more than 0.0046 should produce. Includes a correction to how divergences are attributed. The surviving value belongs to the last stage to produce *or rewrite* a column, and attributing by producing stage alone manufactures false findings: savings_interest_income and tax_free_savings_income originate in frs_spine and are rewritten by the SPI channel, so a naive pass reports them as raw-mapping divergences outside every signed class — the exact signature that made the water defect real. With rewrites folded in, every beyond-band divergence lands in an established class and the only raw-mapping one is already signed. The queue itself is left unadjudicated. The verdict is defect by construction because the register holds only the two water entries, which is the correct starting state; each class needs a ruling before it becomes an entry, and the E9 derived-benefit gap is the one item that may argue against swapping rather than for signing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Twin determinism passes: two independent full builds are payload-identical across every store key, differing only in bytes because HDFStore stamps write times. All four identity receipts pass the property they exist to prove — that recomputation is invariant to row order. e4, e5 and e6 nevertheless report matches_stored_columns false, and the cause is the instrument, not the spine. _frs_only_frame scopes the survey channel by excluding household_is_spi_synthetic alone, which was right when the SPI channel was the only layer stacking rows. E8 added two more: the capital-gains incidence clone and the 270 band donors. Those rows carry values copied from their sources, so recomputing an identity-keyed draw for a clone's own household id disagrees with a stored value that was never drawn for that id. The measurement is unambiguous: excluding all three flags leaves exactly 16,288 households, the raw FRS count, which is the scope the #723 receipts ran at and passed. e8's own receipt passes because it recomputes the clone and donor logic explicitly instead of assuming unstacked rows, and e4's mismatch list is entirely identity-keyed draw columns — precisely those a clone inherits rather than draws. Recorded as unsigned and unfixed: the scope must exclude every stacked layer, the fix belongs with the e7 receipt work since both are ladder maintenance, and until then these three results say nothing about the spine and the L1 leg of the gate is not satisfied. Also noted: e5 and e6 report the failure with an empty mismatch map, which is not actionable evidence and should name what disagreed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Building a spine that carries every stage through E8 and running the whole ladder surfaced that e4, e5 and e6 had gone stale. Each was written when the SPI support channel was the only stage stacking rows; E8 added the capital-gains clone and the CGT band donors after them. The failures are in the instrument, not the spine, and the experiment that proves it is the unchanged tool against two artifacts: the pre-E8 spine passes e6, the post-E8 spine fails it. Three mechanisms, which is why one fix did not cover them. e4 recomputes identity-keyed draws and a stacked row carries a value copied from its source, never drawn for its own id. e5's regional uprating scales to a per-region mean over the frame's owner households, so stacked rows move the denominator — confirmed by running it both ways on one artifact. e6 fails scoped as well as unscoped: its NHS allocation normalizes against an absolute budget, so it needs stage-time weights, and it divided out the SPI channel's share while the clone's mass_split went unrestored. Scoping now excludes every stacked layer from one declared flag list, and the weight divisor reads its factors from the declared operations instead of hardcoding them. The divisor is driven by the flags the artifact actually carries rather than by the committed roster. The first implementation used the roster and divided the clone factor out of a spine built before that stage existed, skewing the comparison the other way; the pre-E8 artifact caught it at once. Both vintages now pass. Recorded at the helper: a new mass-redistributing op kind has to be registered there, and the failure mode if it is not is a receipt silently comparing against the wrong grossing scale. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two gaps the whole-spine parity screen surfaced, both in already-merged increments rather than in new work. The spine never mapped healthy_start_vouchers, free_school_breakfasts, free_school_fruit_veg or free_school_meals, so parity reported three of them as columns the candidate does not produce. The incumbent maps them straight off the person tapes in create_frs, which makes this an E2/E3 omission rather than deferred work — #685 is UC deduction attributes, bus fares and WAS debt, none of which touch these. heartval is on both the adult and child tapes and the three school columns are child-only, so adults read zero rather than propagating NaN, which the test pins. Measured against the reference the three contract columns land at -0.000089, -0.000001 and +0.000008; free_school_breakfasts is not an engine-known variable so it never enters the 145-column surface. The identity ladder had no e7 check at all — the increment that introduced row stacking, and so the one whose interaction with E8 broke e4, e5 and e6, was the only one without a receipt. It now receipts the support-channel layer: each entity's channel and clone index, the composite source key, and the propagation of a household's channel to its persons and benefit units. Bitwise on both surfaces, since these are labels and integer indices. A negative control confirms it is not vacuous: flipping one channel label fails it and names the column. It deliberately excludes the employer_pension_contributions = 3 x employee_pension_contributions derive. That is a real E7 layer, but E8's salary_sacrifice rewrites the multiplicand in place afterwards and the relation survives on only 95.9% of survey-channel persons, so asserting it would fail for the wrong reason — the #721 rewrites-provenance class. The coverage manifest is regenerated for the new source-manifest hash (145 required, 0 exclusions, unchanged) and the spec bundle sha re-pinned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The branch now sits directly on main rather than on the #623 stack, so every derived value is recomputed from its producer against that base: the UK spec bundle sha, and the release input-coverage manifest's source-manifest hashes. source_stages.json and spec/sources.yaml are rebuilt from main's copies with the four in-kind outputs re-applied to both, so the lockstep projection agrees. The four gate-battery mirrors in the data shard were already correct at this base and are unchanged. Coverage stays 145 required, 0 exclusions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The living adjudication packet is a Claude artifact, which is private by default — a reviewer clicking through from the PR hits an access wall. The packet therefore also lives in the repo, where GitHub renders it in the review itself and it travels with the branch. Same content as the interactive page: what the shares measure and their limits, the three incumbents, per-class share tables with donor or raw-source truth, the levels table with the taxable-to-taxable SPI comparison, coverage, diagnostics, and the open items. Adds the UKDS EUL clause 11-12 citations and the disclosure-control statement, which apply wherever these aggregates are posted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
MaxGhenis
added a commit
that referenced
this pull request
Aug 23, 2026
…_VERSION #747 records the uk-data release tag beside the byte identity (SOURCE_VERSION / source.version). A regeneration for another release must not inherit the committed tag, so the identity now carries the tag (from --release, or --version with explicit pins), the in-memory patch sets it, and the on-disk move is anchored to the SOURCE_VERSION assignment — never a global replacement of a tag literal, which also appears in unrelated pins such as the registry-parity pinned_version. The lockstep test binds the committed reference's source.version to the tool's constant once it exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
juaristi22
added a commit
that referenced
this pull request
Aug 23, 2026
An armed UK national build crashed before its first stage: _source_pins stored the full Ledger provenance block under the 'ledger_facts' role, but role_pins_digest requires each role to carry exactly sha256 and size_bytes, so every armed run raised ValueError at input_pins_digest. The two halves landed independently — the strict pin contract with Logbook adoption (#666) and the Ledger role pin with the compile-parity wiring — and neither PR branch fires it alone, because only an armed run (--ledger-facts) reaches this path and the licensed runs were held. It reproduces on both #743 and #747 branches and on main. The pin now carries the feed's verified digest and byte size; the richer Ledger identity block already travels in safe_artifacts, source_vintages, and the diagnostics build block, so nothing is lost. This fix belongs upstream in #743 (the run-readiness PR whose runbook documents the armed command); it rides the build branch until then.
juaristi22
added a commit
that referenced
this pull request
Aug 23, 2026
Calibrates a WS-E spine at national level with the #622/#623 seam, without the certified-input replay: the spine already carries the SPI and CGT derivations as declarative source stages (E7/E8 ported the same pinned sources, QRF stages, reviewed fences, and conservation receipts), so re-running the June-convention replay would discard the spine's certified derivation and substitute an equivalent one. Reuses the harness pieces that are posture-independent: sha-pinned inputs, Ledger register compilation at the declared calibration year, the frozen national doctrine, the validated H5 writer, US-format v6 diagnostics with the uk block, and a Logbook spool row. Scores against the incumbent on the same frozen register before the diagnostics are written, so the score block rides the diagnostics in a single pass (no two-pass sha dance). Assessment posture, stated in the build record: non-certified input, the declared battery is not enforced; the numeric fences that are honestly measurable here (target fit, weight ratio, ESS, zero-weight strata, admin anchors) are evaluated against the thresholds declared in uk/gates.json, and every non-evaluated entry is listed with its reason. The spine's twelve structural NaN columns (SPI-channel concepts undefined on the base-FRS channel) are zero-filled from a fixed allowlist with a receipt; any NaN outside the allowlist aborts. Adjudication (Maria, 2026-08-23): the June convention is transitional and retires with the completed swap; gate integration for the spine posture is follow-up work on the #747 lane.
UC had no dedicated row despite being what the armed calibration binds. Measured modeled universal_credit through the engine on both artifacts: unweighted the spine carries 9% more UC-positive benunits than the incumbent at an equal per-recipient level, and the reported column is exact against the raw tab to every digit. The halved weighted caseload is the calibration boundary itself, not a spine defect — the incumbent's weights already embody a UC caseload target since 1.56.15, so the comparison is before-medicine to after-medicine. Recorded watch-item: would_claim_uc frozen at 0.55 (U8, uk-data#452), the pre-registered lever if the armed run's UC fit is strained. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Whole-spine parity instruments and evidence for #686 (WS-E E10, epic #665). This PR delivers the comparison layer that decides whether the microcosm-built spine can replace the incumbent, plus two fixes the comparison itself surfaced. It deliberately does not perform the swap: per the sequencing ruling, nothing that runs the national driver or moves gate semantics starts until the spine's evidence is reviewed, and the swap/retirement work is a follow-up PR.
Instruments
enhanced_frs_2024_25.h5carried uk-data#461 (raw benunit table no longer sernum-ordered → every benunit-level variable misassigned). The parity screen compares nonzero shares, which are permutation-invariant, so it is structurally blind to this defect —is_marriedreads a byte-identical 0.256587 on both artifacts while every value sits on the wrong benefit unit. Re-pinned to 1.56.16 (adjudicated; identity corroborated three ways), all instruments and the four gate-battery mirrors re-cut. Reference-side movement ≤0.0046 per column, so the ±0.02 screen is undisturbed. The spine itself was never exposed:frs_spinehas sorted the benunit table by id since its original ingest commit.CWATAMT/CSEWAMT; the incumbent's fill-after-addition NaN-zeroes the charge for all 1,663 Scottish households (the whole +0.1009 divergence — incumbent-side, reported upstream as policyengine-uk-data#467). Our own level was also short (CWATAMTDis water-only); both the spine mapping and the council-tax netting now share one helper addingCSEWAMT1at the household's own observed discount factor. Verified at every rung: share unchanged at 0.878377, Scottish level ~£395/household vs ~£185 before.uk/spine_swap_signed_differences.json+ loader): the migration-scoped record of adjudicated spine-vs-incumbent differences, seeded with the two water entries. Strict vocabulary, unique ids, precise per-surface scoping, no expiry (permanent adjudications; expiring suppressions stay in the per-gate exclusion registers). Advisory, not a gate — the standing health checks are the existing battery (support bounds, tail concentration, input-mass parity, aggregate-admin anchors, take-up signal, enum domains), per the US precedent; this register documents the one-time swap decision.tools/verify_uk_spine_parity.py— diffs a candidate extraction against the committed reference on entity counts, nonzero shares, and (optionally, licensed) weighted totals; every difference must match a register entry or the verdict isdefect. Fenced against manufactured verdicts: the reference side is never derived from the candidate, a candidate claiming the incumbent's sha is refused, and--strictfails unused register entries.build_uk_efrs_parity_reference.py --candidate-h5— extracts the same-shape surface from a candidate spine with the same producer (same engine, aliases, rounding), structurally unable to write the committed reference.compare_uk_h5_payload.py --structure-only— additive verdict mode for the eventual control-vs-candidate comparison: structural predicates stay strict, value differences must be signed.--check e7receipts the support-channel layer (the one stacking increment that had no receipt); a negative control proves it non-vacuous. e4/e5/e6 had gone stale against E8's stacking — three distinct mechanisms (identity-keyed draws on copied rows; a population-dependent regional-uprating mean; the NHS budget normalization needing stage-time weights) — fixed with scoping driven by the artifact's own stacking flags and mass factors read from the declared operations. Green on both the post-E8 and pre-E8 spines.free_school_meals,free_school_fruit_veg,healthy_start_vouchers,free_school_breakfasts): a raw-mapping omission from the merged E2/E3 port, previously misattributed to E9. Coverage is now 145/145; the three contract columns land within 0.0001 of the reference.Evidence (licensed,
data/ukds/acceptance/686-spine-swap/; receipts inexperiments/686-uk-spine-swap-receipts.md)rewrites, not justproduces), with the only raw-mapping divergence (water) already signed.Comparison ledger (adjudication packet): committed in this branch as
experiments/686-uk-spine-comparison-ledger.md— GitHub renders it directly. An interactive, continuously updated version lives as a Claude artifact (private by default; ask María for access). The committed rendition carries the UKDS EUL clause 11–12 citations.Honest state
verify_uk_spine_paritycurrently reportsdefect— by construction, not as a spine-quality verdict. The 26-column queue (E5/E6/E7/E8 classes, entity counts, two net-new columns) awaits adjudication in this PR.required_at_buildseam re-point, CGT injection retirement,macro_realism, threshold arming, and the candidate-name move topopulace_uk_2024.Refs #686, #736, #145, #665. Upstream: policyengine-uk-data#467.
🤖 Generated with Claude Code