Skip to content

Swap the microcosm-built UK spine into the national build and retire the incumbent feed #757

Description

@juaristi22

Parent: #145 (workstream E of the UK-migration epic #665). Successor to #686. Reconciled 2026-08-25 with the #623 calibration-seam plan after the first armed calibration campaign (9 runs on the uk-first-calibrated-run-623-686 build branch).

Scope, and what moved out

This issue owns the spine lane after #747 merges: the defect batch the calibration campaign surfaced (A), the per-step gating regime for the spine build (B), and the W9–W11 retirement plus dead code/markdown cleanup (C).

Moved out to #623 (PR #743): the calibration-only driver. The new production calibration seam (tools/calibrate_uk_national_dataset.py + calibration_seam.py) is the "national driver becomes calibration-only" step — it takes whichever h5 it is given, materializes measures, solves weights under doctrine, and never modifies a data variable. What stays here is the other half: retiring the legacy June path (tools/build_uk_national_dataset.py, with its SPI re-derivation and imputation stages) once the seam is proven, which is spine-lane cleanup per the 2026-08-24 ruling.

Gate: nothing here starts until #747 is reviewed and merged. Every item either modifies the spine build (invalidating #747's licensed evidence) or retires a fallback the comparison depends on.

State at the point of handover

leg state
L0 build + determinism complete — 3 rungs, twins payload-identical, record identity exact at 52,846
L1 identity receipts e4–e8 green on both the post-E8 and pre-E8 spines
L2 whole-spine parity signed_parity — 26 beyond-band divergences all signed, 0 unsigned, --strict clean
L3 baselines measured — 177 input-mass totals, 47 QRF tail grids
First armed calibration loss 0.079 · 378/388 bindable · candidate beats incumbent 293–48 on the frozen register · fiscal totals (UC, income tax) still 10–18% low

A. Spine defect remainder

Two of the five landed in #747 rather than waiting for a rebuild here — the review made the case that fixing the spine before the swap is cheaper than certifying it twice:

Still open here:

  1. Spine sidecar self-pin. spine-c.build.json carries the full stage census, artifact/resource pins, seeds and rules-engine version — and no reader joins it today. Downstream consumers should inherit provenance from the sidecar rather than re-deriving it; wire the join and pin the h5 to its sidecar.
  2. Support-channel gaps: slc.repayments.england_postgraduate has zero support on the spine; the top-income (£1m+) channel holds 2 records on both datasets. Each needs a measured support channel or an explicit adjudication that the gap is accepted.
  3. Investigation, not yet a code fix: fiscal totals 10–18% low on the calibrated candidate increasingly looks like spine income distribution rather than target defects. Assessment lane; findings feed back here or into a successor.

B. Per-step gating regime for the spine build

The 2026-08-24 ruling: no imputation or spine gating in the calibration path — the seam runs a calibration-scoped battery only. That is only coherent if the spine build polices itself, stage by stage, the way the US builds do (#327 doctrine):

  1. Per-stage imputation-health gates. Each imputation stage gets a gate consuming its own receipts (support bounds, incidence vs donor, conservation) so a regression is caught at the stage that caused it, not at a whole-spine diff. The signed-differences register stays the swap-decision record; the per-step gates are the standing health checks.

  2. Wire the dormant CGT donor support bounds — or decide they cannot be terminal. cgt_band_donor_support_bounds.json is declared in country_package.json and read by nothing; the other four support-bounds resources (was_wealth, lcfs_consumption, etb_vat, etb_services) are each listed in the uk_support gate's support_bounds_resources and in battery_bindings.py. The CGT one is not an oversight — the resource records why in its own semantic_fit_note: its intervals describe donor-stage initial support, and E8's hmrc_cgt_gains_spine Table-3 redraw moves amounts onto a different band surface afterwards, so "terminal support-gate applicability requires reviewer confirmation".

    That makes this the sharpest argument for per-step gating rather than a stray wiring chore. The other four work as terminal gates because nothing rewrites their columns after they clip to donor; CGT is the one stage where a terminal gate provably cannot express the constraint. Two ways out, and the choice is the adjudication: gate the donor stage at stage time, before the redraw, on the bounds as they stand; or derive post-redraw bounds from the Table-3 surface and gate terminally like the others. Either way the resource stops being declared-but-dead. (This also closes the third of the three sign-offs left open on Port the CGT structure, salary sacrifice, and student loan stages onto the UK spine at FRS 2024-25 #740.)

  3. Re-home the family-coverage contract. uk_release_family_build_stages requires 12 required_at_build families; the spine driver builds 10, the June driver 2, so neither driver alone can pass — the W9 seam. Fix by pointing the guard at the spine plan that actually builds the families (the spine's hmrc_spi_income_spine / hmrc_cgt_gains_spine supersede the June driver's two). Never by relaxing the gate, and never by sidecar plumbingfamily_coverage status='required_at_build' is an executable contract, not documentation.

  4. Arm the thresholds against the spine's own line (US Export input-mass-parity gate references the raw base, flagging correct calibration gains on PUF-imputed columns #327: a parity gate compares a release to its own previous certified line, not a foreign artifact): flip uk_input_mass_parity's reference to the spine's first certified release; arm uk_qrf_tail_concentration from the Assemble the microcosm-built UK spine: whole-spine parity and swap acceptance #686 L3 baselines (47 QRF tail grids, 12 household); RC battery + macro_realism on the swap build. Every armed threshold records which run measured it.

  5. Release-cut certification. The microcosm-data release-verification contract expects the full 21-entry battery on a published artifact — that expectation maps to the monolith. Combining spine-side per-step gating + the seam's calibration-scoped battery into a publishable certification is contract evolution; until it lands, seam-built records say shippable: false with the reason.

C. W9–W11 retirement, dead code, markdowns

  1. Retire the legacy June path once the seam reproduces the campaign result (the v9-equivalent live proof in the Calibrate the UK national build from Ledger-backed targets #623 plan): tools/build_uk_national_dataset.py's SPI re-derivation, replace_uk_spi_support_tables's zero-weight-channel refusal, and the CGT redraw all encode the June convention the spine has superseded.
  2. W8 — certified-identity registry entry for the spine artifact. W11 — retire the CGT injection path the incumbent needed.
  3. UK_CANDIDATE_DATASET_NAMEmicrocosm_uk_2024 (uk_runtime/terminal_gates.py:72, currently populace_uk_2023). Ruling 2026-08-25: the candidate carries the microcosm name, not the legacy populace one — superseding the earlier populace_uk_2024 note; the repo itself has moved to PolicyEngine/microcosm, and the parity-instrument tests already use microcosm_uk_2024.h5 as the canonical candidate filename. Sweep the stray populace_uk_2024* fixture names in test_uk_rowwise_build_driver.py in the same pass. Check UK_REFERENCE_DATASET_NAME alongside: once the spine is the build, "reference" means the previous certified microcosm line, not the enhanced-FRS artifact.
  4. Un-pin the incumbent from microcosm-data/contract.py:219 and uk_runtime/weighted_integrity.py:241 once nothing reads it as an input. The parity reference itself stays — it is the historical comparison instrument and keeping it is how the swap remains auditable.
  5. Dead code: UKPolicyEngineAdapter (zero production callers — the harness rebuilt its capability outboard); the assessment harness tools/calibrate_uk_national_spine.py retires once the seam driver covers it.
  6. Markdown dispositions — a committed .md is a claim nobody re-checks; promote to a gate/test, make regenerable, or archive with a date:
file disposition
UK_COVERAGE_PROGRESS.md stale and misleading — baselined on the enhanced-FRS 2023-24 artifact; live content already lives in release_input_coverage_manifest.json. Delete.
experiments/686-uk-spine-comparison-ledger.md the adjudication packet; make regenerable so it cannot drift from the register beside it
experiments/686-uk-spine-swap-receipts.md R0–R5 run receipts. Archive as dated history once the swap lands
experiments/630-uk-gate-adjudication-receipts.md check each threshold it justifies is pinned in gates.json with provenance, then archive
experiments/612-uk-carrier-payload-receipt.md the "should be a test" case — confirm compare_uk_h5_payload.py covers it, then delete
CONTRACT_FINDINGS.md #462 findings; the fix is merged and gated. Delete

Sweep the same pattern outside the UK lane (COVERAGE_PROGRESS.md, PROGRESS_BUILD*.md, _698-*.md, _*LANE-NOTES.md, FINAL_REPORT.md) as a separate cross-lane PR.

D. Acceptance

  • A: the spine self-pins its sidecar and resolves or adjudicates both support-channel gaps. (The NaN and age-tail fixes landed in Whole-spine parity for the UK: re-pinned incumbent, comparison instruments, signed register, and the E10 evidence (#686) #747; its licensed evidence and parity receipt are re-measured there.)
  • B: every imputation stage gated on its own receipts, including the CGT donor stage, whose support bounds are either wired at stage time or re-derived post-redraw rather than left declared-but-dead; the family-coverage guard passes against the spine plan unchanged; uk_input_mass_parity references the spine's own certified line and uk_qrf_tail_concentration is armed from L3, each with run provenance.
  • C: the national build runs from the microcosm spine with no policyengine-uk-data artifact as an input; the June path and the CGT injection are retired; the candidate is microcosm_uk_2024; no committed Markdown asserts an invariant not also enforced by a test or gate.
  • The whole-spine parity instrument still runs green against the frozen reference, as the historical record of what the swap changed.

Carried forward, not blocking

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions