Baseline integrity: does the certified world carry the law legislated into its future? (#99)

[vahid-ahmadi]

Closes #99. Branches from main — independent of the other UK PRs.

The repo has a baselines registry (#13) that describes worlds, and ingests that stamp which world a run executed. Nothing checked that the certified policyengine-uk world actually contains the measures already legislated into the years it computes at.

That gap is quiet by construction. A counterpart at 2027 or 2028 measured against a baseline missing an announced measure diverges for a reason no axis in #59's registry names — so it gets attributed to whatever axis the analyst reaches for, and the real cause never appears.

This PR was rewritten. Version 1 reported two measures as missing. Both were wrong — guessed parameter paths that failed to resolve, read as proof of absence. DTrim99 caught the first in review on #100; searching the tree for the rest turned up the second. What follows is the corrected registry, and the changes that make this class of error hard to repeat.

Six measures, probed against a real policyengine-uk 2.89.2

verdict	measure
carried	two-child limit abolition	2 in 2025, unbounded from 2026
carried	income tax threshold freeze	flat 12,570 / 37,700 through 2028
carried	property income tax rates	April-2027 rise present: 0.20→0.22, 0.40→0.42, 0.45→0.47
carried	High Value Council Tax Surcharge	full banded schedule at gov.hmrc.council_tax.high_value_surcharge.amount
consistent	fuel duty freeze extension	no parameter encodes it — checked another way
absent	pension tax-free lump sum	no lump/commencement/tax_free_cash/pcls node in the tree

The honest headline is now the opposite of version 1's: the certified world carries the announced law. That is a weaker story and the correct one. The one real absence is a live PolicyEngine-UK development item, not a scoring blocker for 28 October.

The two-child result remains a bonus finding: it confirms the registered pre_ab2025 world is a genuine counterfactual and not a restatement of the baseline — which #49's case battery and #67's decomposition both lean on.

What the two false gaps changed about the probe

The failure was not carelessness about two paths; it was a probe that could not tell three different things apart.

• A reading is now scalar | subtree | absent, never "missing." Version 1 collapsed subtree-and-absent into one bucket. gov.hmrc.income_tax.rates.property is a subtree — it did not return a number, and that was read as "not there" when it was the rise itself.
• absent requires a NAME SEARCH of the whole tree, not one path that failed. A guessed path proves nothing. Both false gaps would have died at this rule.
• Every reading records its exact path and date, and a re-probe replays them verbatim. Version 1 reconstructed paths from key names, which was independently a source of false readings.
• corrections_log keeps both errors in the file, with how each was caught. A registry whose job is refusing unearned excuses cannot quietly delete the two it published.

The same rule now guards #100's scoreability registry, where the identical failure mode exists in a form --resolve cannot catch.

CI

.github/workflows/baseline-probe.yml installs the pinned engine, runs --probe, and fails on any drift between the committed registry and what the engine actually returns. Weekly schedule, so an engine bump that drops a measure surfaces on its own rather than at the next Budget.

[vahid-ahmadi]

Review request — @MaxGhenis @DTrim99.

This one is part of a batch; the whole queue, with a suggested merge order and what is blocked on whom, is in #104 so you can triage in one place rather than PR by PR.

[DTrim99]

Verdict: request changes. The headline claim — that this verifies the certified world contains the law legislated into its future — isn't enforced by anything that runs in CI. What CI actually executes checks the registry against itself.

Critical

• pipeline/check_baseline_integrity.py → reprobe() is the only function that queries the live policyengine-uk engine, but it's gated behind --probe, is never called by any test, and CI (pytest tests/ -q, no engine installed) never invokes it. Everything CI runs (validate + the tests) reads the registry's own recorded probe numbers and asserts them against hardcoded literals — a registry-against-itself tautology. The missing verdicts (the whole point) are author-asserted, never measured; probe: {2028: null} is not confirmed against the engine. To make the check real, wire a CI job that installs the pinned engine and runs --probe (e.g. on a schedule) — otherwise it can't catch the regression it exists to catch.
• data/uk/baseline_integrity.json + the PR body cite data/externals/obr-policy-effects.json (#75) as the evidence for the consistent fuel-duty verdict, but that file does not exist on main. The consistent verdict rests on an absent/unmerged artifact, and the test only asserts the string -0.129pp appears in prose — evidence is text, not a computed check.

Should

• baseline_integrity.json uses bare Infinity literals (lines ~20, ~29). Python's json.load accepts them, but the file is not valid JSON — any strict/JS consumer on the app/ side will fail to parse it.
• pipeline/ is a new top-level dir; all existing pipeline code lives in scorecard_db/. ci.yml lists pipeline in ruff targets, but the module isn't imported by build_db or fed into the no-drift build — effectively dead outside its own test.
• The reprobe() pin guard raises SystemExit if the installed engine ≠ 2.89.2; combined with never running in CI, the engine path will bit-rot silently once the pin advances.

🤖 review via Claude Code

[vahid-ahmadi]

Rewritten, and the PR body with it — it was advertising a headline I've retracted.

Your catch on #100 was the same defect as this registry's two missing verdicts, so I re-probed all of them. Both were wrong. gov.hmrc.income_tax.rates.property is a subtree carrying the April-2027 rise; the surcharge lives at gov.hmrc.council_tax.high_value_surcharge.amount with the full banded schedule, not at the path I guessed. The first bracket's amount is 0 because it's the below-threshold band — that's what v1 mistook for an empty schedule.

So the finding is now the opposite of what it was: the certified world carries the announced law. One genuine absence remains (pension tax-free lump sum), and it's an upstream development item.

The probe changes matter more than the two entries. v1 could not distinguish a path that doesn't exist from a path that exists and isn't a scalar — it collapsed both into "missing," which is exactly how a subtree became an engine gap. Readings are now scalar | subtree | absent; an absent verdict requires a name search of the whole tree rather than one path that failed; every reading records the exact path and date, and a re-probe replays them verbatim instead of reconstructing paths from key names.

Both errors stay in corrections_log with how each was caught. A registry whose whole purpose is refusing unearned excuses shouldn't quietly delete the two it published.

On your note that pipeline/ should exist — it's already on main, six modules. If you were seeing it absent, that's likely the stale-DB trap: data/scorecard.db is gitignored since #74 and survives branch switches, so rm data/scorecard.db before running the suite. I re-verified main clean at 285 passing.

[vahid-ahmadi]

CI is green now. The failure was mine and unrelated to the findings: this workflow probes the sibling registries too, and those validators (validate_budget_2026_registry.py from #100, validate_cgt_reform_spec.py from #102) don't exist on this ref yet, so the step failed on a missing file.

Each is now skipped when absent and starts enforcing the moment its PR merges. The skip is announced via ::notice:: rather than being silent — a gate that quietly validates nothing reads as a gate that passed, which is the thing #74 and #95 were about.