UK counterpart compute pipeline (managed sims, pe_gap-degrading registry)

[vahid-ahmadi]

Part of #40 — the standing UK compute stage, structured to the US pipeline's binding contract. (Not "Closes": first execution under simlock and the country promotion through build_comparison.py remain — both need the managed environment, which this machine doesn't have.)

What's here

• pipeline/compute_uk_counterparts.py — mirrors compute_counterparts.py: managed_microsimulation(country="uk") on the certified populace-uk bundle under the sim lock; emits weighted counts and GBP aggregates only (data/pe/pe_uk_metrics.json + pe_uk_meta.json with per-variable metadata), so every rate stays derived downstream in one place. Microdf auto-weighting rules honored: boolean-array subgroup cuts, map_to for cross-entity, no weight-array touching.
• Two runs: baseline (calibrated take-up) and fullpart (take-up input-variable overrides forced on — the policyengine_uk_inputs framework in the DB's ReformRef vocabulary), giving the entitled-population denominators the DWP take-up comparison needs.
• Coverage mapped to the five ingested UK lanes: DWP take-up benefit-unit caseloads/expenditure (GB-restricted, NI split via the country variable), HBAI poverty counts BHC/AHC by age group + population denominators, HMRC taxpayer counts and liability aggregates, OBR per-benefit spending lines, UKMOD shared instruments.
• tests/test_uk_pipeline_registry.py — registry-consistency tests runnable anywhere; the module imports engine-free (the policyengine import is inside build_sim).
• README gains the UK reproducing block.

The honesty mechanism

Engine variable names drift across policyengine-uk versions, so every concept resolves through an ordered CANDIDATES registry — first existing variable wins and is recorded in pe_uk_meta.json. A concept with no resolvable variable emits a pe_gap row instead of crashing: the gap stays on the page per the status taxonomy, and the run log lists each unresolved concept for a one-line registry fix. Reviewers should treat the candidate lists as the first thing to correct against the current engine — that's exactly what they're for.

Honest status

⚠️ Not executed on this machine — no managed policyengine.py environment or certified populace-uk bundle here. What's verified: the full repo test suite (135 passed) including the new registry tests, ruff format, and engine-free import. First execution under simlock is owed by whoever holds the managed environment, and its pe_gap/unresolved log becomes the follow-up work list.

Pairs with #48 (the claims these counterparts will attach to).

🤖 Generated with Claude Code

[vahid-ahmadi]

Review pass on this file against compute_counterparts.py (the US reference) found six ways it could emit wrong numbers under status: "ok". All six are fixed in 30933b7. Candidly: the script has still never been executed — no policyengine-managed environment or certified populace-uk bundle here — so these are corrections by inspection against the documented contract, not observed failures.

#	Defect	Fix
1	Fullpart sized its override array with len(sim.calculate(flag, YEAR)) — a calculate before set_input, which caches baseline take-up and makes the override a no-op	Sizes from sim.populations[entity].count; nothing is calculated on the sim until every override is set
2	Overrides hardcoded to the annual period	Set at each flag's own definition_period — all 12 months for monthly variables, eternity fallback. UK benefit variables are commonly monthly, so a year-only set_input misses the periods the engine reads
3	No verification the toggle took effect	flag_means_after recorded per flag, plus a hard abort if no override applied, if none reads back as on, or if fullpart is identical to baseline across every take-up-sensitive value
4	Missing flags skipped silently (if flag in vs:)	flags_missing / flags_failed recorded and warned; a run with zero applied overrides raises instead of emitting a fullpart series
5	in_pov.count() if hasattr(...) else len(in_pov) — an unweighted ~10⁵ record count emitted as a poverty-rate denominator with status: "ok"	Weighted path requires a genuine weighted MicroSeries (.weights present); otherwise pe_gap. An unweighted number is never presented as weighted
6	in_relative_poverty_ahc fell back to the generic in_poverty — a wrong-concept value under status: "ok"	Removed. Audited the rest: pays_higher_rate no longer lists higher_rate_earned_income_tax (a GBP amount, not a boolean). The invariant "candidate lists hold only genuine synonyms" is now stated in the file and enforced by tests

Plus: YEAR = int(sys.argv[2]) → parse_year(sys.argv) reading argv[1] (there is no argv[2] here — the US script's argv[1] is its run-selection list; script.py 2026 silently ran 2025), invocation documented in the docstring and README.

Defect 3 was the worst of them: if all five take-up overrides silently no-op — very possible, since policyengine-uk take-up is parameter/seed-driven and these input variables may not exist — fullpart came out byte-identical to baseline, so every DWP take-up denominator was the baseline caseload and every take-up rate would have printed as ~100%, with every row reading ok. That is now three separate hard failures, not a warning.

Still unverified until the first real run

• The engine entry point. from policyengine import managed_microsimulation; managed_microsimulation(country="uk") did not match the binding contract, which is pe.us.managed_microsimulation() in compute_counterparts.py and docs/ARCHITECTURE.md. It is now pe.uk.managed_microsimulation(), mirroring that surface, with an explicit UNVERIFIED comment on the line — I could not confirm the UK accessor's spelling from anything in this repo and did not want to guess silently.
• Which CANDIDATES names exist in policyengine-uk, and therefore how many concepts land as pe_gap.
• Whether the FULLPART_OVERRIDES input variables exist at all. If UK take-up is purely parametric, the fullpart run will now abort loudly on first execution and the take-up lane needs a reform-based approach instead — which is the correct outcome, and the point of the change.

Tests: extended tests/test_uk_pipeline_registry.py to 21 tests covering no-cross-concept-fallbacks, BENEFIT_LINES → CANDIDATES resolution, argv index, definition-period selection, the fullpart movement assertion, and a denominator test proving an unweighted count can never be emitted as ok. Full suite 148 passed / 4 skipped; ruff format --check clean; the module still imports without policyengine installed. Beyond the committed tests I also drove the whole file end-to-end against a throwaway fake engine (in scratch, not committed) to confirm the set-before-calculate ordering, the monthly/eternity period handling, and that both abort paths actually fire — which is evidence the control flow hangs together, not evidence about the real engine.

[MaxGhenis]

Dual-gate review: NEEDS-FIXES — fix rather than hold (verified independent of #48). Blockers:

1. First managed run cannot complete: all five take-up override names are absent from the current PE-UK vocabulary — the engine's inputs are would_claim_pc, would_claim_housing_benefit, would_claim_uc, would_claim_child_benefit, claims_all_entitled_benefits (the set the compute campaign's fulltakeup runs used successfully). The code reaches its own hard abort.
2. Entity/geography mis-mappings: ESA/JSA are benefit-unit variables and winter fuel is household-level, but BENEFIT_LINES maps them person/person/benunit — awards project onto wrong units and overcount; geo_masks() maps household-valued country to every group entity and silently emits whole-UK totals labeled GB on failure.
3. Full-participation validation is global: any one sensitive value moving blesses the run — UC moving can mask unchanged PC/HB denominators (the silent 100%-take-up failure class). Validate per benefit.
4. Coverage vs advertisement: pension-age HB emitted as unrestricted housing_benefit; HBAI numerators without matching denominators/cuts; HMRC marginal-band candidates declared but only all-taxpayer aggregates emitted.
5. Provenance: read sim.policyengine_bundle (US precedent in compute_counterparts.py) instead of only data_bundle/dataset.
6. Weighting contract: weighted_population() reads series.weights directly, violating the file's own never-touch-weight-arrays rule — MicroSeries.count() is the weighted count.

Verified sound: nested engine import, map_to usage elsewhere, set_input-only fullpart mutation, simlock docs.

[vahid-ahmadi]

All six blockers fixed in 6d77d5a. Blocker → fix:

#	Blocker	Fix
1	takes_up_* names absent from the engine	FULLPART_OVERRIDES is now exactly the campaign-proven set: would_claim_pc / would_claim_housing_benefit / would_claim_uc / would_claim_child_benefit / claims_all_entitled_benefits, pinned by a test so a drift back to guessed names fails CI
2	Entity/geography mis-mappings	esa_income/jsa → benunit, winter_fuel_allowance → household (pinned). geo_masks() rebuilt: person-level country projection + first-person anchor per group entity (the first_person_anchor_state rule from the US script — household-enum projection to group entities crashes in core). New restricted_mask(): a GB/NI line whose entity has no anchor emits pe_gap — the silent whole-UK-labeled-GB path is structurally gone, and the refusal is unit-tested
3	Global movement check	Per benefit: each of pension_credit, housing_benefit, housing_benefit_pensioners, universal_credit, child_benefit must move on its own or the run raises naming the stuck benefit; guarantee_credit/savings_credit are warn-only components (SC is closed to new claims). The UC-moves-but-PC-doesn't case is now a committed failing test
4	Coverage vs advertisement	Pension-age HB emitted under the DWP external program id housing_benefit_pensioners (head-SP-age benunit cut, same anchor rule as geography), alongside — never instead of — the unrestricted housing_benefit line the obr/ukmod lanes use. Every HBAI numerator cut now has its matching subgroup population denominator (total/children/pensioners). HMRC emits taxpayer_count + tax_liability per marginal band (basic_rate/higher_rate/additional_rate, the external adapter's subgroup ids) through boolean band concepts; the declared-but-unused total_income/marginal_tax_rate_band candidates are removed
5	Provenance	sim.policyengine_bundle read first (US precedent), stringified like the US script; data_bundle/dataset only as a fallback, with bundle_source recorded so a fallback can never masquerade as the bundle record
6	Weighting contract	weighted_population() identifies a genuine MicroSeries by type and returns MicroSeries.count() — the weighted count. Zero .weights access, enforced on the AST by a test

Suite: 158 passed / 4 skipped (was 148); ruff format --check clean.

Honesty note, same as before: still zero real executions — no managed environment here. The band flag candidates (pays_basic_rate_tax etc.) and the is_SP_age anchor are name-candidates that resolve-or-pe_gap on first run; the entry-point spelling caveat from the previous comment stands unchanged.

[DTrim99]

Review — UK counterpart compute pipeline

Faithfully mirrors the US compute_counterparts.py contract. I checked the weighting discipline specifically: no manual weight multiplication, no .sum() on unweighted arrays, no cross-entity aggregation without map_to — subgroup cuts are boolean-array indexing of the MicroSeries with weights preserved. Determinism holds (append-ordered rows, insertion-ordered meta, no timestamps in the JSON, default=str stable → byte-stable), and take-up is reversible (baseline and fullpart are separate sims with del sim; gc.collect(); overrides set on the fullpart sim only, before any calculate). Test coverage below the engine boundary is strong.

Should address (both cheap, before the first managed run)

• fullpart override is a Python list, not an ndarray (compute_uk_counterparts.py:471, arr = [value] * n) — the US path uses np.ones(n, dtype=bool) (compute_counterparts.py:117). set_input on some core versions expects an ndarray; a list may coerce oddly or fail silently. Use np.full(n, value) / np.ones(n, dtype=bool) for parity.
• year param is half-wired. Every sim.calculate reads the module-global YEAR (lines 566/686/703/773), but build_sim accepts a year param that compute_run never passes (line 669). It works because main() sets global YEAR, but a caller passing a non-global year would silently compute on YEAR. Tighten or drop the param.
• dict(bundle) (line 447) assumes the bundle is mapping-like, where the US path reads .items() — type-guard it, or it raises inside build_sim if the record shape differs.

Note

The engine entry (pe.uk.managed_microsimulation()), the FULLPART_OVERRIDES names, and which CANDIDATES resolve are genuinely unrun here (first execution is deferred to the managed env) — self-documented, and build_comparison.py still needs to learn the UK row shape (the status field) before promotion. Flagging so first-run risk is known.

Reviewed with Claude Code assistance.

[vahid-ahmadi]

All three should-address items fixed in b54444c:

Item	Fix
Override is a Python list	arr = np.full(n, value, dtype=bool) (US parity with np.ones(n, dtype=bool)); numpy imported inside build_sim so the module stays engine-free-importable. Pinned by a test asserting the list form is gone
year param half-wired	Dropped entirely — build_sim(fullpart) reads the module-global YEAR like every sim.calculate in the file, with the rationale in its docstring. Signature pinned by test
dict(bundle) assumes mapping	Read via .items() only when the record is mapping-like; a truthy non-mapping record is stringified and labeled bundle_source: "policyengine_bundle (non-mapping, stringified)" instead of raising inside build_sim

Suite: 160 passed / 4 skipped (was 158); ruff format --check clean. The Note items (entry-point spelling, first-run resolution, build_comparison.py learning the UK status field before promotion) stand as known first-run work.

[DTrim99]

Re-review — all addressed ✅

All three with matching regression tests:

• fullpart override is now np.full(n, value, dtype=bool) (ndarray parity with the US np.ones), pinned by test_fullpart_override_is_an_ndarray_not_a_list.
• half-wired year param dropped — build_sim(fullpart) only; test_build_sim_has_no_half_wired_year_param locks the signature.
• dict(bundle) is now hasattr(bundle, "items")-guarded with a non-mapping fallback recorded in bundle_source.

The engine path (managed_microsimulation, CANDIDATES, FULLPART_OVERRIDES) is still unrun, as expected/self-documented. Looks good.

[DTrim99]

Re-reviewed — all three findings fixed with matching regression tests (np.full ndarray, year param dropped, bundle type-guard). Approving.

[MaxGhenis]

Re-gate round 2 — the five names are right, but the combined world isn't

The literal take-up vocabulary is now correct (all five verified as annual benunit variables in pinned 2.89.2), and the earlier entity/provenance/weighting fixes hold. But the dual gate's second pass — this round read the pinned engine source directly — found the full-participation construction itself doesn't survive contact with the engine's mechanics. Five blockers, each with the engine evidence:

1. The fullpart run invalidates Housing Benefit. Forcing all five flags true in one simulation drives HB toward zero, not all-entitled: in pinned 2.89.2, housing_benefit_eligible requires ~would_claim_uc (UC/legacy mutual exclusion — there's an engine regression test for exactly this) and housing_benefit_reported > 0, i.e. eligibility is receipt-gated, so a take-up world cannot create entitled non-recipients for HB at all (verified in variables/gov/dwp/housing_benefit/housing_benefit_eligible.py). HB needs its own construction — or an emitted pe_gap with a citable link. Because the current validation accepts any movement including decreases, the wrong direction passes silently.

2. Pension Credit component rows are wrong-concept status=ok. Engine guarantee_credit/savings_credit are entitlement components that never consult would_claim_pc — only aggregate pension_credit applies the claim flag (verified: the only would_claim_pc reference is in pension_credit.py). DWP's claims are claimed GC and savings_credit_only (which additionally requires guarantee_credit == 0 — see sources/dwp-takeup/adapter.py:88); the run emits raw components as recipients/spending under a savings_credit slug the closed registry doesn't recognize.

3. Per-benefit validation is too weak to catch either of the above. It aggregates both sensitive metrics and passes when either moves in any direction — a probe with frozen PC recipient_count but moved spending passed as 1/2 moved, which lets a broken caseload denominator (and hence a 100% take-up error) survive. Needs per-metric, direction-aware checks.

4. The pe_gap path bypasses descriptive gate App design: public scoreboard + mission control, four views #9. Gap records carry no claim id, rationale, or action link, and nothing imports pe_uk_metrics.json into the DB — the exporter only surfaces claims that already have a DB result, so a missing capability can still disappear entirely. Concrete instance: when either HBAI poverty variable is unavailable, the loop emits only the total-numerator gap and continues — an empty-registry probe produced 2 HBAI gaps where 12 claims were owed. Post-The database leaves git: derived artifact, built in CI, published to Supabase storage #74 the citable-link rule is enforced through db.diagnose(); plain PEResult(status=pe_gap) never reaches it.

5. Attachment needs to be source- and unit-aware against the merged foundation. Rows omit source and unit concept, but the closed registry deliberately keeps DWP benefit_units distinct from UKMOD families, and only savings_credit_only exists as a program — one generic benunit count can't serve both lanes. Also the three HMRC boolean band variables targeted don't exist in the certified engine (it exposes the tax_band enum instead), so current runs guarantee uncited band gaps.

Two smaller items: in a real managed environment the focused tests go 31 passed / 2 failed (the fake MicroSeries is rejected when real microdf is installed); and the managed outputs aren't in build_db.py or tracked, so the new no-drift gates won't see their first creation — worth deciding deliberately where they live post-#74.

Holding #76–#80 (the campaign runs) on this — the minimum repair is benefit-specific fullpart construction, claimed-GC / SC-only logic, per-metric direction-aware validation, and an exhaustive source-aware registry where every value lands as a result or a cited gap through the post-#74 DB path. Happy to pair on the HB construction question if useful — it may genuinely be a pe_gap (the receipt-gating means the engine cannot express HB entitled-non-recipients today), which would be an honest and citable outcome, not a failure.

🤖 Generated with Claude Code

[vahid-ahmadi]

All five blockers plus the two smaller items addressed in 6d1e794. The engine reads were the useful part of this round — thank you for going into 2.89.2 directly.

1. Housing Benefit. Taking your suggestion: it is a cited gap, not a number. housing_benefit_eligible requiring both ~would_claim_uc and housing_benefit_reported > 0 means eligibility is receipt-gated, so a take-up world cannot create entitled non-recipients for HB and forcing the flags on drives it toward zero. HB's fullpart rows now emit pe_gap with that reason and an upstream link; the baseline rows are unchanged. HB is out of TAKEUP_VALIDATED_PROGRAMS, because demanding movement would force the wrong-direction number straight back in.

2. Pension Credit components. Correct — the only would_claim_pc reference is on the aggregate, so the raw components published a wrong-concept value under ok, under a slug the closed registry does not recognise. PC_COMPONENT_LINES now emits CLAIMED guarantee credit and savings_credit_only, both gated on would_claim_pc, with savings-credit-only additionally requiring guarantee_credit == 0 (adapter.py:88).

3. Validation. Now per metric and direction-aware. Every comparable (program, metric) pair must INCREASE — forcing take-up on can only add recipients and spending — a fall is a hard failure, and a program missing one of the two sensitive metrics is a partial pass and also fails. Three tests reproduce your probes, including the frozen-caseload-with-moved-spending one that used to read as "1/2 moved".

4. Cited gaps. pe_gap now requires a registered reason from GAP_REASONS, each carrying a rationale and an action link, and an unregistered reason raises. The HBAI loop is exhaustive: hbai_gaps() emits all six cells per housing-cost basis, so the empty-registry probe now produces the 12 owed rather than 2.

5. Source- and unit-aware rows. Every row carries the external source id and unit concept through ROW_IDENTITY, so DWP benefit_units and UKMOD families stay distinct, and an unattributed (program, metric) raises. The band cuts resolve through the engine's tax_band enum first — you are right the three booleans do not exist at the pin, so leading with them guaranteed three uncited gaps every run — with the booleans kept as fallbacks.

Smaller items. The MicroSeries helper now builds a real one where microdf is importable and the type-identity stand-in only where it is not, so the focused tests hold in the managed environment too. And the output location is decided deliberately: data/pe/, tracked, alongside the US pe_metrics.json — these are run outputs an operator commits and a later build reads, so their first creation is a normal commit rather than something the no-drift gate must be taught about. The ingest that reads them lands with #76, which is why every gap is cited at emission here.

Suite 294 passed, ruff format clean. Still unexecuted against the managed environment, and knowingly so — happy to hold #76–#80 until you have re-read this.

[vahid-ahmadi]

Follow-up: I resolved the whole CANDIDATES registry against a real installed policyengine-uk 2.89.2 rather than reasoning about it, and it surfaced one defect in my own fix plus confirmation of your engine reads.

The defect (fixed in 3d73b91). An EnumArray stores integer INDICES; decode_to_str() is what turns them into member names. The fallback path I wrote compared the undecoded array to "BASIC" — matching nothing and silently emptying every band cut, which is the same failure class as the uncited band gaps this finding was about. A series that cannot be decoded is now not used at all (the band falls to its boolean fallback and then to a cited gap), and the comparison accepts the member name or its value so a core release changing which one it decodes to cannot empty the cut either.

What the check confirmed:

• Exactly three concepts in the registry are unresolvable, and they are the three boolean band flags — your read exactly, and the reason tax_band now leads. Its members really are BASIC / HIGHER / ADDITIONAL.
• housing_benefit_eligible reads, verbatim:

  already_claiming & (social | lha_eligible) & ~claiming_uc & (capital <= limit)

  where already_claiming = housing_benefit_reported > 0 and claiming_uc = would_claim_uc. Receipt-gated and UC-exclusive, precisely as you described — so the cited gap is the right call, not a convenient one.
• guarantee_credit, savings_credit and would_claim_pc are all annual benunit variables, and the only non-test reference to would_claim_pc outside its own definition is in pension_credit.py. So the components genuinely never consult it, and the new gating maps at the right entity.
• Every BENEFIT_LINES entity matches the engine's own variable entity.

Suite 296 passed. Still unexecuted against a managed bundle — but the parts that were reasoned about are now checked.