ONS effects of taxes and benefits (mode 1, distributional): the decile-incidence population HMT publishes only as charts (#90)

[vahid-ahmadi]

Closes #90. Stacked on #91 (uk/thinktank-ingest) — both touch uk_aliases, relationships and build_db, so this targets that branch rather than main; review #91 first.

The lane #61/#68 went looking for and could not have. HM Treasury publishes its decile impacts as unlabeled chart bars — 132 marks, zero digitizable values, which that lane declares rather than digitizes. ONS publishes observations, so this one carries numbers: 2,640 claims across the five income concepts, one per stage of the tax-benefit system.

original  →  gross  →  disposable  →  post-tax  →  final
          + cash     − direct      − indirect    + benefits
          benefits   taxes         taxes         in kind

Source reconciliation, asserted by the build

5,280 published observations = 2,640 claims + 2,640 deliberate drops

The dropped half is the Deflated value series. Its price base is stated nowhere — not in the dataset, not in its CSVW metadata, not in the value-deflation code list — and a real-terms figure whose base year is unknown cannot be reproduced or compared with anything PolicyEngine computes.

The 2024 bulletin does name CPIH excluding Council Tax. But that is a different release from this version-3 dataset, and attributing its deflator here would launder one publication's methodology onto another's numbers. So the nominal half is emitted and the real-terms half is a declared, tallied drop.

Two identity facts, read out of the data rather than assumed

The time dimension is genuinely mixed. ONS's own code list is named financial-and-calendar-years and it means it: 1977–1993 are calendar years, 1994-95 onward are financial. Each row carries the basis its label implies, and a financial year keys its END year — so a calendar-1990 observation and a financial-2020-21 one never share a time basis.

Five income concepts are five quantities, not one with a qualifier. The same quintile, statistic and period under two concepts are two different claims; a test asserts their claim ids differ.

Coverage rides on the claim

Each row carries pe_expressibility:

concept	PE-UK	why
original, gross, disposable	expressible
post_tax	partial	PE-UK has VAT and the duties, but no indirect-tax aggregate — a counterpart must sum heads explicitly
final	not_expressible	benefits in kind (health, education, housing subsidy) have no PE-UK counterpart

The not_expressible rows carry an action_link (gate #9), so a final-income divergence can never be reported as an engine defect when it is a coverage gap — the same call the HMT lane made about public services.

Provenance

Pinned to dataset version 3, not latest_version: ONS's bulletin series runs later than the dataset, so a silent follow would change the observations under the claims. The CSV and its CSVW metadata are both vendored with SHA-256 pins.

raw/README.md records what was surveyed and not kept — including the decile table this issue originally asked for, which turns out to be a 2015-saved .xls from a discontinued series. Quintiles are the honest current grain, and the lane says so rather than implying decile coverage it does not have.

ONS ranks by equivalised disposable income — not HBAI's BHC/AHC ranking, not UKMOD's, not RF's — so the quintiles are registered per source and recorded DISTINCT from all three.

Verification

• suite 307 passed
• two builds agree on content_hash
• no-drift gate clean
• ruff format --check clean

Reviewers

@MaxGhenis @DTrim99 — the call I'd most like challenged is dropping the deflated half. The alternative is to carry it with the price base recorded as unknown, but I think an unreproducible real-terms number is worse than an absent one. Second: quintiles, not deciles closes this issue with less granularity than it asked for; I'd rather state that than vendor a decade-stale file.

[DTrim99]

Reviewed closely — checked out the branch, recomputed the pins, and ran tests/test_ons_etb.py (21 pass) against the real vendored CSV. Same discipline as #91; the "reconciliation asserted by the build" approach holds up. Stacked on #91 — this targets uk/thinktank-ingest, so it should merge after #91 lands. Approving, with one small tidy and two hygiene notes.

Verified against the real data

• Accounting identity 5,280 = 2,640 claims + 2,640 drops is computed and enforced (raises if it doesn't close), and reconciles from the CSV. The dropped half is exactly the Deflated value series, dropped because the price base is stated nowhere — and the code correctly does not launder the 2024 bulletin's CPIH deflator onto a v3 dataset. Good call.
• Time basis — calendar (1977–1993) vs financial (1994-95+) is read from ONS's own financial-and-calendar-years code list; a financial year keys its END year, and parse_period validates suffix == start + 1 and raises otherwise. 17 calendar + 27 financial = 44 periods checks out; a calendar-1990 and a financial-2020-21 row never share a basis.
• 5 concepts = 5 quantities — claim_ids are value-invariant and differ across concepts, with a collision-raising uniqueness gate. Tested.
• Expressibility — original/gross/disposable expressible, post_tax partial (no PE-UK indirect-tax aggregate), final not_expressible (benefits in kind), and the not_expressible rows carry the action_link so a final-income divergence can't be read as an engine defect. Correct.
• Provenance — pinned to dataset v3 (no /latest anywhere; tested), and both the CSV and its CSVW metadata carry SHA-256 pins that are recompute-and-checked before parsing (I recomputed both — they match). Adapter parses by column name (not order), with closed registries that raise on any unknown concept/quintile/statistic/deflation/geography.
• Determinism — content_hash/claim_id are sort_keys canonical (dict-order invariant), no timestamps into the hash, and data/lanes.json is byte-identical to the app/public mirror.
• Quintile-not-decile honesty — quintiles registered, and raw/README.md records the discontinued 2015 .xls decile table as surveyed-and-not-kept, with no implied decile coverage. Nicely honest.

One thing to tidy (non-blocking)

The DISTINCT prose overclaims. source.json, adapter.py, and uk_aliases.py comments all say ONS quintiles are distinct from "HBAI's… not UKMOD's, not RF's" (and IFS), but the only DISTINCT edges (and tests) that actually exist are ons_etb ↔ ukmod and ons_etb ↔ resolution_foundation — there's no ons_etb ↔ hbai or ons_etb ↔ ifs edge. It isn't a silent-aliasing bug (the canon key is per-(source, axis), so cross-source values can't unify without an explicit alias, and none exist), but if the DISTINCT set is meant as the assertion registry that proves the claim, it currently reads as half-populated. Either add the ons_etb↔hbai/ifs edges (+ extend the DISTINCT test) or soften the comments to name only the pairs you actually register.

Hygiene notes

1. adapter.py's open(..., newline="") has no explicit encoding="utf-8" — benign here because the SHA-256 pin fixes the bytes, but worth pinning the encoding so a platform default can't ever change how it reads.
2. Consider a .gitattributes forcing LF (-text / text eol=lf) on the vendored .csv/.csv-metadata.json — I hit a spurious SHA-pin mismatch locally from Windows autocrlf converting line endings on checkout; the committed blob is correct under LF, but a Windows contributor running the suite would see it fail confusingly.

Strong lane — the "ONS publishes observations, HMT publishes chart bars, so this one carries numbers" framing is exactly the right complement to #68.

Reviewed with Claude Code assistance.

[vahid-ahmadi]

Thank you — all three addressed, and the .gitattributes one especially: better you hit it than a contributor.

The DISTINCT prose overclaim. Correct, and I fixed it in the direction the evidence points rather than by padding the ledger. The ons_etb edges are now exhaustive against the vocabularies that exist — all five UKMOD quintiles, both RF quintiles, and the IFS decile boundaries.

But not against HBAI, deliberately, and the prose now says why: HBAI registers no decile or quantile vocabulary at all (its subgroups are children/pensioners/working_age/total), so there is nothing on that side to assert against. Adding ons_etb ↔ hbai edges would have been the same overclaiming with the sign flipped — a pair naming a value nobody registered. I took your "soften the comments to name only the pairs you actually register" option for HBAI and your "add the edges" option everywhere the other side genuinely exists. The adapter docstring's stray "not HMT's" is gone for the same reason. A test pins the absence and the reason.

Hygiene 1 — encoding. Pinned encoding="utf-8" on the read, with a comment noting the SHA gate fixes the bytes but not how they are decoded.

Hygiene 2 — line endings. Added .gitattributes with -text on *.csv, *.csv-metadata.json and *.jsonl. This is the better fix than documenting the workaround: the vendored artifacts are pinned by SHA-256, so their bytes have to survive checkout unchanged on every platform, and a spurious pin mismatch is about the most confusing failure this lane could hand someone.

This branch also carries the #91 review fixes via merge, so it stays mergeable behind it.

Suite 310 passed, two builds agree on content_hash, ruff format clean.