OBR policy costings (mode 2): registry, offline certified compute, staging, descriptive comparison

[MaxGhenis]

Builds the lane in #54 (Max's steer, 2026-08-16: center PE model development on the Scorecard — compute PE counterparts to OBR's published scores first, add machinery only when a comparison needs it). Static half; #55 is the macro half.

What's here

• data/uk/obr_measure_reforms.yaml — measure → reform registry, 26 measures (3 expressible / 18 partial / 5 not_expressible), each with the verbatim OBR description, a policyengine-uk parameter dict (or null), construction, per-head mapping to PE variables, and notes on what is partial. Every parameter path resolves against the installed pe-uk 2.89.2 tree; the PA/HRT freeze reverses to Table 3.19's published with-indexation PA/HRT path (assumption-matched to OBR's own counterfactual; the PE HRT parameter is a band width, handled).
• pipeline/compute_uk_obr_costings.py — offline-hardened managed sims on the certified bundle (populace-uk-2023-dd68c73-4aa4b14-20260619T023711Z, SHA-256 verified against the release manifest before any sim), plain-dict reforms, year-major with baseline reuse, per-head GBP aggregates in OBR's positive_gain_to_exchequer convention, measure orientation (reversal_on_certified_world ⇒ measure Δ = −(reversal − baseline); the literal reversal delta is kept in the artifact), per-run JSON artifacts, --dry-run (registry validation, no sims), --restage (re-derive staging from artifacts, no sims).
• results/uk/staged/obr_costings.jsonl — 20 rows under the campaign ingest contract (external_claim_match descriptor exactly as the harvest carries it, engine version, full bundle id, status/construction/run_id/annotations).
• pipeline/compare_uk_obr_costings.py + results/uk/obr_costings/COMPARISON.{md,csv} — descriptive rows: measure × FY, OBR, PE, signed ratio, bin, named axes (static vs behavioural-adjusted; certified-world vs announcement baseline; CY-proxies-FY; head scope), remainder unexplained. No score-like summary statistic; benchmark_class = different_model throughout.
• tests/test_obr_costings_registry.py — 33 cases (registry schema, engine path/variable resolution, NICs head aggregate, sign and orientation identities for both constructions and channels, staged-row contract, artifact provenance trace, ambiguous-source refusal, forced offline mode, output-free dry run, comparison bins at every boundary). Suite: 200 passed.

Smoke run (6 measures × FY2026-27/2027-28, certified bundle, policyengine 5.0.2 / pe-uk 2.89.2)

GBP bn, positive = gain to the Exchequer. Descriptive bins, not scores.

Measure	FY	OBR	PE	PE/OBR	Bin
PA and HRT freezes (EFO 3.17)	2026-27	+34.010	+47.681	1.40	1.25–2
PA and HRT freezes	2027-28	+38.475	+53.452	1.39	1.25–2
Additional-rate threshold reduction (3.17)	2026-27	+0.940	+1.840	1.96	1.25–2
Additional-rate threshold reduction	2027-28	+0.970	+1.933	1.99	1.25–2
SB2024 employee NICs main rate −2pp	2026-27	−9.129	−11.965	1.31	1.25–2
SB2024 employee NICs main rate −2pp	2027-28	−9.244	−12.247	1.33	1.25–2
SB2024 HICBC threshold/taper	2026-27	−0.641	−1.721	2.68	≥2
SB2024 HICBC threshold/taper	2027-28	−0.647	−1.844	2.85	≥2
AB2024 employer NICs package	2026-27	+23.610	+16.247	0.69	0.5–0.8
AB2024 employer NICs package	2027-28	+24.027	+16.422	0.68	0.5–0.8
AB2025 remove two-child limit	2026-27	−1.887	−1.104	0.59	0.5–0.8
AB2025 remove two-child limit	2027-28	−2.101	−1.182	0.56	0.5–0.8

Axes named on every row; measure-specific ones worth flagging: HICBC — PE's child-benefit claiming is fixed in a static run, so the reform-induced claiming OBR costs does not move (a concrete PE-improvement candidate surfaced by the comparison); employer NICs — income-tax + NICs heads only, no Employment Allowance/firm mechanics (partial by construction); the personal-tax gaps sit where OBR's behavioural adjustments bite hardest (additional rate ≈ 2×). Everything beyond the named axes is unexplained.

Diagnostic (not staged): PA 12,570 → 13,070 for 2026 = income tax −£4.475bn static.

Not in this PR

• DB ingest of the PMD claims — rides UK externals → DB ingest path (16,924 claims, five sources) #48's vocabulary; ingest_campaign.py still needs an OBR-aware attachment seam (deliberately rejects UK family obr today). Stage first, ingest second, as the US campaign did.
• The full 26-measure population run — after review; smoke was 798 s / peak RSS 32.9 GiB for 13 sims on this machine.

Sol-built (offline lane) then Sol clean-context reviewed; the one construction defect found in the first pass (literal reversal deltas staged un-oriented, every bin opposite_sign) was fixed and restaged from artifacts before this PR.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review — verified by running the branch locally (head 9a7b09d)

How verified: ran the exact CI commands in a clean worktree, the full suite with and without PyYAML, --dry-run, a registry consistency script over obr_measure_reforms.yaml, and a trial merge of current main.

Finding 1 — CI is green only by runner-image luck: PyYAML is an undeclared dependency

tests/test_obr_costings_registry.py:25 (and pipeline/compute_uk_obr_costings.py:38, compare_uk_obr_costings.py:32) hard-import yaml, but CI runs uv run --with pytest pytest with no PyYAML anywhere in the repo. The job passes on this exact SHA (run 32075753535, 306 passed) because ubuntu-latest preinstalls PyYAML in the base interpreter uv picks up. On any machine without that — e.g. uv-managed CPython on macOS — the same command dies at collection:

tests/test_obr_costings_registry.py:25: in <module>
    import yaml
E   ModuleNotFoundError: No module named 'yaml'
!!!!!!!! Interrupted: 1 error during collection !!!!!!!!

So a runner-image update silently reds the whole pytest job, and no contributor can reproduce CI locally. Cheapest durable fix: add --with pyyaml to the pytest line in ci.yml — that also un-skips the yaml-gated tests the #67/#68 lanes had to importorskip for exactly this reason (both of those registries shipped JSON to stay CI-safe; declaring the dep once lets future lanes use YAML honestly).

Observation (by design, worth stating)

--dry-run hard-requires policyengine-uk (compute_uk_obr_costings.py:2952-2954) — deliberate, since the dry run's value is resolving every parameter path against the installed pe-uk tree, but it means the advertised "registry validation, no sims" cannot run on unmanaged machines at all; the schema half lives only in pytest. Fine as-is given the tests, just not obvious from the flag's help text.

Verified sound

• Suite: 306 passed / 14 skipped with PyYAML present — matches the CI run byte-for-byte; ruff format clean.
• Registry: 26 measures, zero duplicate measure_keys, triage exactly 3 expressible / 18 partial / 5 not_expressible as the PR body states; every expressible measure carries a pe_reform dict.
• Orientation identities are real tests, not tautologies: test_measure_orientation_for_both_constructions_and_channels parametrizes both constructions × both channels with hand-specified expected (literal_delta, pe_value) pairs, and unknown channel/construction raise.
• Staged contract: results/uk/staged/obr_costings.jsonl is exactly the 20 rows claimed; COMPARISON.csv carries 26 rows.
• No rebase risk: a trial merge of current main (post #48/#70–#73) completes with zero conflicts.

The smoke-run divergences this lane surfaced are already being decomposed in #67 (per-measure axes registry; the employer-NICs row is the worked example), so the two lanes join up as intended.

[MaxGhenis]

Gate round 1 — clean compute hygiene; six findings, three blocking

What passes cleanly: offline/local-only enforcement, engine/bundle/data pins on the compute side, serial one-at-a-time simulation, signed descriptive comparisons with forced constructed status, and no wins language.

1. Blocking — announcement-time baseline honesty is absent. The 26-measure registry carries no baseline-world identity; compute creates one reform=None managed baseline per year and reuses it across five fiscal events, and claims/artifacts/staged rows all lack baseline_policy and an executed baseline_key — so the cross-baseline guard cannot do its job. Each fiscal event needs its provenance-backed registered world (the pre_obbba_law / hmrc_indexed pattern), executed, mirrored into claims, stamped on results, with both match and mismatch tests. Never default these events to current_law — that exact defect just blocked OBR policy effects → DB ingest: 266 claims, four macro-effect metrics #81 too.

2. Blocking — the feature disappears on a fresh CI build. Since The database leaves git: derived artifact, built in CI, published to Supabase storage #74, data/scorecard.db is built from scratch every CI run by scorecard_db/build_db.py — an ingest absent from that chain vanishes on rebuild, and PROGRESS.md states this PR doesn't modify scorecard_db at all. The 621 claims and 20 results need a registered ingestor with the standard shape: one transaction (delete-by-owned-source + insert + register_baselines_txn + lane), drift-gated exact accounting, post-commit feed sync, rollback/idempotence tests — plus care on source ownership, since the existing UK ingestor deletes source='obr' wholesale.

3. Blocking — reconciliation isn't repository-reproducible. The committed README records workbook hashes but neither OBR workbook nor the extraction adapter is vendored — the full-harvest test reads ~/scorecard-harvest and skips in CI, so nothing can verify checksums, full-precision reads, or account for the 24,937 exclusions between the 25,558 source rows and the 621-row slice. Vendor the primaries and a deterministic adapter with exact selected/drop tallies (the repo must be self-sufficient — the The database leaves git: derived artifact, built in CI, published to Supabase storage #74 rule).

4. High — closed identity unfinished. Eight retained condition keys absent from STANDARD_CONDITIONS (costing_phase, fiscal_event, impact_channel, line_item, note, sign_convention, spending_head, tax_head), aliases unregistered, 228 spending claims on an unsupported exchequer_impact, and FY-START claim periods against the DB's FY-END convention (the UK externals → DB ingest path (16,924 claims, five sources) #48/UC deductions: FRR family re-harvested from primary sources + DB ingest #52 rule: claims key the end year; engine start-year is result provenance).

5. High — relationship hard-coded. Every claim copies held_out instead of deriving fail-closed from the canonical registry with the pe-uk-data@dd68c73 evidence.

6. Cleanup: regenerate the 13 artifacts through the current before/after dataset-hash path (drop legacy_artifacts_without_dataset_hashes); the rebase has a .gitignore conflict to resolve preserving both entries.

🤖 Generated with Claude Code

[vahid-ahmadi]

Flagging a CI issue on this branch that isn't about its contents.

This PR is running an older workflow than main's, so its green tick means less than it looks. GitHub runs a pull_request workflow from the PR's own branch, and this one forked before #74 — so it still has the pre-#74 ci.yml, which has neither the determinism check (two builds must agree on the content hash) nor the no-drift gate (git diff --exit-code). It also still tracks data/scorecard.db, the committed database #74 removed.

I found this auditing all 16 open PRs after hitting it on two of my own (#49 and #64, both since fixed). Four branches were affected; this is one of the two remaining.

The fix is mechanical and nothing about your own changes needs to move:

git checkout obr-costings-mode2
git merge origin/main
rm -f data/scorecard.db          # gitignored since #74
PYTHONPATH=. python -m scorecard_db.build_db data/scorecard.db
pytest tests/ -q
git status --short               # must be clean

I haven't pushed to your branch. Happy to if you'd rather I did — say the word.

There's also a structural fix in flight at #95: a gate-freshness check that runs from the base via pull_request_target, so a stale head can't skip the gates in future. It reads main's ci.yml to derive what to require, rather than hardcoding it, so gates added later are enforced automatically.

Worth prioritising this one over #69: #67 consumes this branch's COMPARISON.csv, so its decomposition inherits whatever this build produces.