UC deductions: FRR family re-harvested from primary sources + DB ingest

[vahid-ahmadi]

Part of #39 (staging handoff from #21). Stacked on #48 (base branch uk/db-ingest-uk-externals — it builds on that PR's Metric vocabulary); merge #48 first and this rebases to a clean diff.

Why a re-harvest

#21 staged the UC-deductions claims at ~/scorecard-harvest/uk_deductions/ — machine-local, not vendored. Rather than transcribe from the issue text, the Fair Repayment Rate family is re-harvested from primary sources, vendored in-repo, and page-verified (2026-08-14):

• sources/harvest-uk-deductions/frr/autumn_budget_2024.pdf — FRR passages at paras 2.30 (p.47), 4.111 (p.110), 5.134 (p.142)
• autumn_budget_2024_policy_costings.pdf — verified absence: no FRR costing line in all 93 pages, consistent with UK UC deductions: external scores staged (FRR, JRF floor, historical cap costings) #21's finding that the measure is a financial transaction with no PSNB impact
• DWP press release 2025-04-30 (effective date, 25%→15%)

claims_staged.jsonl carries 8 claims, each with a verbatim quote and page locator: PSNCR +£385m (2029-30), 1.2m gaining households, £420/yr average gain, 700k families with children — HMT and DWP statements staged as separate citable claims — plus the 2.8m households-with-deductions pre-FRR level. VERIFICATION.md records the full audit and what remains un-staged from #21 (JRF floor with its pre/post-FRR baseline_key split, historical cap costings, screening counts).

The ingest

scorecard_db/ingest_uk_deductions.py, under the harvest fail-loud contract:

• The PSNCR line lands as a new cash_requirement_change metric with fiscal_measure=psncr — deliberately not revenue_change, so no PSNB-shaped comparison can ever silently join against it. This is UK UC deductions: external scores staged (FRR, JRF floor, historical cap costings) #21's "benchmark against PSNCR, never PSNB" rule made structural, and test_no_psnb_shaped_claims enforces it.
• Reform rows ride policy_ref {"policy": "uc_fair_repayment_rate"} with the cap change and effective date as reform detail; the pre-FRR deductions level is a baseline claim.
• Everything held_out; the DWP quarterly deductions outturn tables (the calibration source PE's parameters consume) are not staged at all — that boundary is the other UK UC deductions: external scores staged (FRR, JRF floor, historical cap costings) #21 rule.
• models.py gains cash_requirement_change / gainer_count / average_annual_gain (the distributional-impact vocabulary of UK fiscal-event documents).

Verification

4 new tests (headline values, PSNCR-not-PSNB, quote+publication on every claim, DB round trip); full suite 137 passed, ruff clean.

The PE comparator from policyengine-uk#1815 (0.85m benunits, £421 mean gain vs DWP's £420) attaches as a pe_results row once its run artifacts are staged with engine/bundle provenance — that, plus the remaining #21 families, keeps #39 open.

🤖 Generated with Claude Code

---

Merge order: this PR hard-depends on #48 (UnitConcept.GBP and the #33 metric vocabulary come from that base) — merge #48 first; the branch is rebased onto it. #65 adds UnitConcept.GBP independently; whichever merges second takes the trivial enum conflict.

[DTrim99]

Review — UC deductions: FRR family re-harvest

Provenance is honest and the structure is clean: 8 rows, no claim_id collisions, imports resolve, and the figures are round headline quotes expanded (m → raw), not derived. Reviewed as this-lane-only (stacked on #48).

Confirm before merge (stacking)

• UnitConcept.GBP doesn't exist on main — ingest_uk_deductions.py:74-76 (UNITS = {"gbp": UnitConcept.GBP, ...}) relies on it, but this diff only extends Metric, not UnitConcept. It's supplied by the UK externals → DB ingest path (16,924 claims, five sources) #48 base, so merge UK externals → DB ingest path (16,924 claims, five sources) #48 first; if for any reason UK externals → DB ingest path (16,924 claims, five sources) #48 doesn't add GBP, the module import raises AttributeError and every test errors. Just calling it out as an unstated hard dependency.

Should address

• Row-3 quote may not be verbatim (claims_staged.jsonl:3, average_annual_gain): "households expected to be better off by £420 a year on average" reads as a reworded fragment of row-2's sentence ("…1.2 million households will be better off by £420 per year on average…") from the same paragraph. Two rows citing one sentence with differently-worded "verbatim" quotes — either confirm UK adapters (mode 1) #3 is a distinct string in AB2024 or point it at row-2's quote.

Suggestion

• The "verified absence" of an FRR costing line (93 pages) is prose-only in VERIFICATION.md; test_no_psnb_shaped_claims only asserts no REVENUE_CHANGE metric, which is a proxy. A machine-checkable marker (a costing_absent: true note or a grep-assert fixture) would make the absence auditable rather than trust-me.

Reviewed with Claude Code assistance.

[vahid-ahmadi]

All three items addressed in b32f8a0:

1. Merge-order dependency now stated in the PR body: UK externals → DB ingest path (16,924 claims, five sources) #48 first (UnitConcept.GBP + the UK harvest → DB ingest path: attach the 48 staged campaign results #33 vocabulary are its base); branch rebased onto the updated UK externals → DB ingest path (16,924 claims, five sources) #48.
2. Row-3 £420 quote confirmed distinct and verbatim: extracting the vendored AB2024 print shows two separate sentences carrying the figure — para 4.111's "This will mean 1.2 million households will be better off by £420 per year on average…" (row 2) and the para 2.30/5.134 wording "…with households expected to be better off by £420 a year on average" (row 3). Not a reworded fragment; a test now pins both strings verbatim (test_420_quotes_are_two_distinct_verbatim_sentences).
3. Costing absence machine-checked (your suggestion): test_frr_costing_absence_is_machine_checked extracts the vendored 93-page costings PDF and asserts zero occurrences of the measure name; VERIFICATION.md points at the test. Both PDF tests importorskip(pypdf) so bare CI skips them and local runs enforce them.

Suite with pypdf: 182 passed / 4 skipped; ruff clean.

[DTrim99]

Re-review — all addressed ✅

• UnitConcept.GBP — correction to my earlier note: this is supplied by the base (UK harvest → DB ingest path: attach the 48 staged campaign results #33), not UK externals → DB ingest path (16,924 claims, five sources) #48, and the diff correctly extends only Metric. So it imports cleanly and isn't a GBP merge-order blocker. My prior attribution was wrong — sorry for the noise.
• Row-3 "verbatim" quote — now a distinct passage (para 4.111 "£420 per year" vs 2.30/5.134 "£420 a year"), with test_420_quotes_are_two_distinct_verbatim_sentences asserting both appear verbatim in AB2024.
• "Verified absence" — now machine-checked (test_frr_costing_absence_is_machine_checked extracts the costings PDF and asserts zero "Fair Repayment Rate" hits).

Clean from my side.

[DTrim99]

Re-reviewed — all findings addressed (GBP comes from #33 as noted, row-3 quote is now distinct + verbatim-tested, absence machine-checked). Approving.