Complete #622: compile the UK target surface from Ledger facts at the 2024-25 line

[juaristi22]

Closes #622: the UK household-sector target surface compiles from Ledger facts at the FRS 2024-25 line (calibration year 2025, per #723), with a compile-parity gate proving the surface against both production truth and the incumbent registry, release-manifest Ledger pins, and CGT single-homed onto references.

Lands on main — #727 and #729 merged beneath it; the Universal Credit caseload activates on the post-chronicle#184 feed. (Terminology note: the ledger→chronicle rename is deliberately excluded and lands as its own mechanical PR after this one — #737.)

What lands

Shared, country-agnostic (microcosm/build/, microcosm/calibrate/):

• ledger_targets.py: period-token collapse for British year spellings (fy2024_25, census_2023 — same-series vintages previously resolved as ambiguity errors); assertion-policy enforcement (observed-only default; allow_source_projection by contract declaration); three declared multi-fact operations — sum, calendar_year_average, latest_plateau (the UK scorecard: populace_uk_2023 vs enhanced_frs_2024_25 vs admin benchmarks (2026) #731 ruling; Max's verified UC series as test vectors) — always partitioned to a single vintage.
• New target_reference_authoring.py: contract + feed + period → typed references + membership classification (fan-out, geography/assertion pins, uprating holds; zero unexplained deferrals).
• New target_materialization.py: the binding interpreter (prepared columns from value_variable + filters + household conditions) and the three counterfactual provider engines (parameter_gated_threshold, baseline_flag_crosstab, input_substitution_counterfactual — the JCT zero-variable pattern, duck-typed over a model adapter).
• gates.py: a parameterized ledger compile-parity gate over the existing ledger_target_registry_parity_report, strict (name, period, value) projection.
• ledger_artifact.py: shared --ledger-facts / sha-pin CLI helpers.
• calibrate/matrix.py: constraint rows now collapse any entity to the weight entity via declared relationships — benunit-grain references (112+ on this surface) previously aborted on any non-1:1:1 frame.

UK surface:

• Contract at 189 declarations: the 85–90 sex bands split into uniform five-year 85_89 + 90_plus tail constraints (the incumbent's six-year terminal band leaves 91+ unconstrained); all five 410-Gone-suppressed incumbent ancestors signed (NICs pair + IT-relief trio); TCL year-surface accounting updated to the April-2025 facts.
• target_references.json regenerated at period 2025 against the post-chronicle#184 feed (107,550 facts): 388 active references (identity / 54 sum / calendar_year_average on the UC caseload / 347 explicit holds) from 417 fan-out candidates; membership shipped as target_reference_membership.json. dwp.uc.households compiles to the CY-2025 mean of the nine 2025 monthly caseload facts — 6,758,889 vs the incumbent's 6,700,000, a signed +0.9% difference. Deferrals: 27 no-fact, 1 multi-fact (Scotland households, signed), 1 signed-excluded (property-income amounts — the incumbent's ×1.9 undercount adjustment traces to policyengine-uk-data#311/Add coverage-universe targets for uninsured and private coverage #230; binding official Table 3.7 values would knowingly calibrate to 10/19 of the incumbent surface; registry: WS-C deferred adjudications and the reconciliation layer #736).
• CGT single-homed: UK_CGT_TARGET_SPECS → emptied shim; totals compile from hmrc/cgt_statistics facts (£65.937bn / 378k, the 2023-24 outturn).
• Two release-blocking compile-parity gates + frozen fixtures: production release populace-uk-2023-dd68c73-… (149 rows @2023) and the incumbent at 12a1e028 (652 rows; 637 effective @2025), every difference enumerated and signed in committed resources.
• tools/build_uk_national_dataset.py adopts the Ledger pin surface (one resolved artifact feeds the pins, the gates, and Add the UK national calibration step over ledger-backed target references #729's calibration stage, adapted to this surface).

Parity receipts (full 107,550-fact feed)

• Fixture A (production @2023): 211 compiled, 12 exact matches, 6 signed drift rows (≤3.5% except household land value 0.913), 131 region-grain rows structurally unmatched (this activation pins country grain; region activation is the contract's declared next step).
• Fixture B (incumbent @2025): 388 compiled — 187 exact matches, 163 signed drift rows (99 UC payment bands, median ratio 0.999; UC family splits +2.6–15% = publication-month choice; the UC caseload's +0.9% calendar-average delta; CGT identity-hold vs the incumbent's uprated £67.73bn, rationale in the resource), 38 ledger-side additions (15 TCL at 2025, 5 salary-sacrifice recoveries, 2 new 90+ bands, …), 287 incumbent rows on the deferred side.

Authoring receipts, the drift tables, and the implementation trail are posted as comments. Deferred adjudications are centralized in #736.

Out of scope

Running a calibration (#729 owns the seam; run-readiness is the narrowed #623); local_targets.py duck-typing retirement (#708); chronicle-side authoring; the FRS 2024-25 ingest itself (#723); the ledger→chronicle rename (#737).

🤖 Generated with Claude Code

[juaristi22]

Authoring-run receipt (regeneration at period 2025)

Feed: merged UK consumer facts, 98,392 rows (chronicle post-#173 sex-split surface, bundle years 2023–2026 deduped on aggregate_fact_key). Generator: tools/generate_uk_target_references.py against the 189-declaration contract.

{"active": 387, "multi_fact": 1, "no_fact_at_or_before_period": 28, "signed_excluded": 1}
candidates (post fan-out): 415   operations: identity 331 / sum 54   explicit uprating holds: 347

Deferral accounting (full classification committed as uk/target_reference_membership.json):

• 28 no-fact — all lacking any fact ≤ 2025, including dwp.uc.households under the resolve-or-defer ruling: it activates automatically (with calendar_year_average by generator policy) once chronicle#184's monthly UC facts reach a re-staged feed.
• 1 multi-fact — ons.population.scotland_households_3plus_children: the contract wants a household-composition fact that does not exist at ≤2025; signed.
• 1 signed-excluded — hmrc.spi.property_income.amount_by_total_income_band: the incumbent applies a ×1.9 undercount adjustment (traced: policyengine-uk-data#311 / Add coverage-universe targets for uninsured and private coverage #230 — SPI covers only taxpayers with liability; HMRC Property Rental Income Statistics show £46.68bn vs SPI ~£24.5bn for 2020-21). Binding official Table 3.7 values would knowingly calibrate to 10/19 of the incumbent surface. Follow-up paths recorded in the rationale: a chronicle package for the rental statistics + declared reconciliation, or a declared value-scaling operation citing Add coverage-universe targets for uninsured and private coverage #230.

Band-structure ruling (María, 2026-08-21): the incumbent's six-year terminal 85_90 sex bands (which leave 91+ unconstrained) split into uniform five-year 85_89 + 90_plus tail constraints — 4 declarations replacing 2, the 90_plus pair signed as new unmapped declarations.

[juaristi22]

Compile-parity receipts (both gates green; every difference enumerated + signed in committed resources)

Fixture A — production truth (populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z, 149 rows, compiled at period 2023):

kind	rows	note
exact match	12
calibration drift (signed)	6	isc 0.995 · pse 1.032 · land 0.913 / 1.010 · Scotland ages ±0.5%
fixture-only (structural)	131	production calibrated region×age rows; this activation pins country grain — region activation is the contract's declared next step for the 18 two-level targets
ledger-only	209 − 18	activation surplus beyond the June surface

Fixture B — incumbent registry @ 12a1e028 / v1.56.16 (652-row union, 637 effective at 2025; frozen extraction via the incumbent's own _resolve_value, no live run):

kind	rows	note
exact match	187
calibration drift (signed)	162	99 UC payment bands (median ratio 0.999, publication-month noise) · 9 UC family splits +2.6–15% (month choice) · 39 ONS population (MYE vintage; the two 85_89 rows carry the declared band-split semantic difference) · 5 salary-sacrifice (HMRC-table facts vs the incumbent's OBR-forecast rows) · CGT: ledger £65.937bn (2023-24 outturn, identity hold) vs incumbent £67.73bn (uprated forecast) — doctrine consequence, rationale in the resource · esa/isc/pse/land/slc small
ledger-only	38	15 two-child-limit (April-2025 facts; the incumbent pins the same numbers at {2026} and drops them at 2025) · 5 salary-sacrifice recoveries (410-Gone CSV, all five suppressed ancestors now signed) · 2 new 90_plus bands · activation surplus
fixture-only	288	the deferred families (no-fact ≤2025 or signed)

Resources: uk/parity_fixture_production_2023.json, uk/registry_parity_fixture_2025.json, uk/ledger_compile_parity_{production_2023,incumbent_2025}_signed_differences.json; regeneration tools committed.

Verification

Full three-package pytest + ruff check green at the tip (independently re-run outside the implementation sandbox); full-feed authoring, parity, and CGT receipts reproduce the numbers above bit-for-bit. The gate battery digests are recomputed from the merged gates.json (this branch carries #729's coverage gate; the battery pins here supersede the stale ones on that branch).

Implementation trail: seven delegated runs with per-run reports; notable defects caught by receipt review along the way — the Fixture-A period mismatch, a drift classifier that flagged value-equal rows, benunit-grain constraint compilation aborting on real frames (fixed entity-general in calibrate/matrix.py), a UC band tokenizer emitting nan_to_nan rows, and Scotland age selectors summing age+male+female facts.

[juaristi22]

Receipts update — rebased onto main, UC caseload activated (post-chronicle#184)

The branch now sits directly on main (#727 + #729 merged beneath it; base retargeted) as five curated commits. The staged feed rebuilt from chronicle main @ 1cab809: 107,550 facts (was 98,392; sha 226358e7…; all four bundle years now valid and year-invariant).

UC activation (the #731-adjudicated operation, first live use):

dwp.uc.households  value_operation=calendar_year_average  period=2025
months: 2025-04 6,380,000 · 2025-05..08 6,600,000 · 2025-09..11 6,960,000 · 2025-12 7,170,000
mean:   6,758,888.89   (Jan/Feb 2026 plateau months correctly outside the year window)
vs incumbent 6,700,000.0 → signed calibration_drift +0.9%

The selector isolates observed_measure.source_measure_id=total_units — the companion deduction-count and share measures do not multi-match. The only membership movement in the whole feed refresh was this activation: 388 active / 27 no-fact / 1 multi-fact / 1 signed-excluded (candidates 417).

Updated parity receipts: Fixture A 211 compiled (12 exact, 6 signed drift, 131 structural region-grain); Fixture B 388 compiled (187 exact, 163 signed drift incl. the UC row, 38 ledger-only, 287 deferred-side). Full pytest (three packages) + ruff green on the pushed tip, independently re-run.

Related filings from review-prep: deferred-adjudications registry #736 (now incl. TCL-2026 retirement and UC element targets), rename issue #737 (lands right after this PR), chronicle facts request PolicyEngine/chronicle#187 (UC element counts + monthly split history).

[vahid-ahmadi]

Automated review pass (Claude Code, low-effort single diff pass — no verification run).

1. packages/microcosm-build/src/microcosm/build/ledger_targets.py:566 — multi-fact provenance metadata comes from the wrong fact. After facts = ...; fact = facts[0], the value for latest_plateau is numeric_values[-1] and for sum/calendar_year_average an aggregate, but _ledger_metadata(fact, fact_key=_fact_key(fact)) and "ledger_resolved_assertion": _fact_assertion(fact) still describe facts[0] — the earliest plateau/partition member. A latest_plateau target compiled from Jan–Jun facts records the June value with January's fact key, period and assertion, so pins/receipts point at a fact that did not produce the value. This looks like the blocking one.

2. packages/microcosm-build/src/microcosm/build/gates.py:181 — signed_differences is typed Iterable but consumed twice: _signed_difference_keys(signed_differences) exhausts it, then the details["signed_differences"] comprehension iterates it again. Pass a generator and the gate drops the keys or reports an empty signed-difference list while signed_difference_count says otherwise.

3. packages/microcosm-calibrate/src/microcosm/calibrate/matrix.py (~205) — np.searchsorted(weight_ids, destination_ids) assumes the weight table is sorted by id. Nothing in Frame guarantees household_id is ascending; on an unsorted table searchsorted returns positions whose ids don't match, valid fails, and the group-level collapse raises "mapped household id(s) are absent from the weight table" for a valid frame. (The raise is marked # pragma: no cover - frame validation guards this, which only holds when ids happen to be sorted.) A weight_ids-indexed dict / argsort lookup is the safe form.

4. packages/microcosm-build/src/microcosm/build/ledger_targets.py:1418 — the "assertion" branch was removed from _selector_candidates with no compat path, so any ledger_selector still carrying assertion hits raise ValueError(f"Unsupported Ledger fact selector field {key!r}."). The UK target_references.json here no longer uses it, but other country packages or externally-authored references pinning assertion: observation will hard-fail compilation instead of falling through to the new assertion_policy handling.

[juaristi22]

Thanks @vahid-ahmadi — all four findings dispositioned in 442c38b3 (plus c5345171 from the adversarial pass that landed just before your review):

1. Multi-fact provenance from facts[0] — confirmed, fixed. Multi-fact compiles now stamp single-fact metadata from the value-representative member (the latest member for latest_plateau, latest-as-representative for sum/calendar_year_average), name the operation, and carry the ordered member keys as ledger_member_aggregate_fact_keys. Receipts regenerated: the UC caseload row keeps its mean (6,758,888.89) and now records resolved_fact_period: "2025-12" with all nine member keys. Regressions cover the plateau and aggregate cases.
2. signed_differences consumed twice — stale against the tip, regression added. The adversarial-review hardening (c5345171) had already reworked the gate to materialize and self-check every signed row (live kind + both values within tolerance; stale signatures fail loudly). A generator-input regression now pins the materialization.
3. searchsorted sorted-id assumption — confirmed, fixed. The collapse uses an argsort-backed lookup, the misleading no cover pragma is gone, and a deliberately shuffled weight-entity frame proves correct aggregation. US suites untouched-green.
4. assertion selector hard-fail — confirmed, compat restored. assertion is a supported selector key again with absent-counts-as-observation semantics (consistent with the new assertion_policy layer). The UK contract's own ban on it stands; externally-authored references compile.

Full three-package pytest + ruff green on the pushed tip, independently re-run; counts unchanged (388 active / 27 no-fact / 1 multi-fact / 1 signed-excluded); parity receipts unchanged (330 / 488 signed rows, all now live-checked).