Skip to content

Minimum wage coverage and bite (mode 1): the first external check on PE-UK's minimum-wage machinery (#88) - #93

Open
vahid-ahmadi wants to merge 2 commits into
mainfrom
uk/lpc-minimum-wage
Open

Minimum wage coverage and bite (mode 1): the first external check on PE-UK's minimum-wage machinery (#88)#93
vahid-ahmadi wants to merge 2 commits into
mainfrom
uk/lpc-minimum-wage

Conversation

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Closes #88. Branches from main — independent of #91/#92.

PolicyEngine-UK carries minimum_wage, minimum_wage_category and the whole gov.hmrc.minimum_wage parameter tree, and there was no external claim about the minimum wage anywhere in the 15,858 UK rows. 119 claims from the Low Pay Commission Report 2025 data annexes:

metric n
minimum_wage_bite 66 per cent of median hourly pay
minimum_wage_coverage 13 jobs at or below the rate
minimum_wage_coverage_rate 40 per cent of jobs

A JOBS unit concept

Coverage counts jobs. One person can hold two and one household several, and ASHE — the survey behind it — is an employer survey of jobs. Mapping that onto persons because the unit already existed would have misstated what the number is. uk_aliases records lpc:jobs DISTINCT from dwp_hbai:persons and uk_hmrc:individuals.

Count and rate stay separate metrics for the same reason poverty_count and poverty_rate do.

The bite denominator rides on the claim

Bite is the rate over ASHE's median hourly wage of full-time workers. The certified PE-UK world is FRS-based — a household survey whose hourly-pay distribution is not ASHE's. So a PE-vs-LPC bite gap is a survey-population difference before it is an engine question.

That axis is in conditions["denominator"] on all 66 bite claims, a bite claim without one is refused at staging, and source.json states the axis must be sized before any engine defect is named. I suspect this is the family's most interesting finding rather than a problem with it.

What is deliberately not read, tallied rather than implied

  • 191 of the 193 main-annex sheets — the report's other figures (earnings distributions, employment effects, international comparisons). Each is its own population decision, not something to sweep in because the file was already open.
  • The 349-row local-authority coverage table. LA geography is finer than any UK geography this repo registers, and the certified engine cannot resolve it either; opening an LA vocabulary for one table would create identities nothing else can join to.

Both are declared drops with reasons and row counts, asserted by a test.

Every claim also says which rate it is about — adult rate, an age-band rate, or all NMW/NLW rates together are three different questions. LPC's "East of England" is aliased once onto the repo's registered "East".

Verification

  • suite 282 passed
  • two builds agree on content_hash
  • no-drift gate clean
  • ruff format --check clean

Reviewers

@MaxGhenis @DTrim99 — two calls worth challenging. (1) JOBS as a new unit concept: the alternative is to refuse coverage counts entirely until a jobs population exists elsewhere, but I think an honestly-labelled job count beats no claim. (2) Reading only 2 of 193 sheets: I'd rather land a small, fully-identified population than a large half-triaged one, but the split is a judgement and the tally is there so you can see exactly what I left.

minimum-wage machinery (#88)

PolicyEngine-UK carries `minimum_wage`, `minimum_wage_category` and the
whole `gov.hmrc.minimum_wage` parameter tree, and there was no external
claim about the minimum wage anywhere in the 15,858 UK rows. 119 claims
from the Low Pay Commission Report 2025 data annexes:

    minimum_wage_bite            66  per cent of median hourly pay
    minimum_wage_coverage        13  jobs at or below the rate
    minimum_wage_coverage_rate   40  per cent of jobs

Two things introduced on purpose.

A JOBS unit concept. Coverage counts JOBS — one person can hold two and
one household several, and ASHE is an employer survey of jobs — so it is
not interchangeable with persons, households, benefit units or families.
Mapping it onto `persons` because that unit already existed would have
misstated what the number is. uk_aliases records lpc:jobs DISTINCT from
dwp_hbai:persons and uk_hmrc:individuals. Count and rate stay separate
metrics for the same reason poverty_count and poverty_rate do.

The bite denominator, on the claim. Bite is the rate over ASHE's median
hourly wage of full-time workers; the certified PE-UK world is FRS-based,
a household survey whose hourly-pay distribution is not ASHE's. So a
PE-vs-LPC bite gap is a SURVEY-POPULATION difference before it is an
engine question. That axis rides in conditions["denominator"] on all 66
bite claims, a bite claim without one is REFUSED at staging, and
source.json states the axis must be sized before any engine defect is
named. This is likely to be the family's most interesting finding rather
than a problem with it.

What is deliberately not read is tallied, not implied: 191 of the 193
main-annex sheets (the report's other figures, each its own population
decision) and the 349-row local-authority coverage table. LA geography is
finer than any UK geography this repo registers and the certified engine
cannot resolve it either, so opening an LA vocabulary for one table would
create identities nothing else can join to.

Every claim also says WHICH rate it is about — the adult rate, an
age-band rate, or all NMW/NLW rates together are three different
questions. LPC's "East of England" is aliased once onto the repo's
registered "East".

Suite 282 passed, two builds agree on content_hash, no-drift clean,
ruff format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me

@DTrim99 DTrim99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on tip f934ddcpytest tests/test_lpc_minimum_wage.py = 20 passed. Clean; same lane discipline as the rest of the stack.

Verified:

  • Provenance — both .xlsx SHA-256 pinned and recompute-checked at build + in a test; edition pinned ("Low Pay Commission Report 2025"), source.json uses versioned GOV.UK asset URLs (not "latest").
  • Adapter — parses by sheet name + title-anchor (a reshuffled workbook raises), no cell-ref fragility. Coverage(jobs) vs bite(percent) modeled as distinct Metrics from distinct columns; spot-checked UK 2025 raw cells (2,020,000 / 6.6) stage verbatim with no re-derivation or %/count confusion.
  • Accountingingest asserts counts == _EXPECTED and finish raises on claim_id collision; 119 = 66 + 13 + 40 pinned, and the 191 unread sheets + 349 LA rows are declared drops with reasons (tallied and asserted), so the identity holds.
  • New metrics MINIMUM_WAGE_BITE/_COVERAGE/_COVERAGE_RATE + UnitConcept.JOBS keep count vs rate apart (poverty_count/rate precedent) and are non-duplicative.
  • Fail-loud on unmapped geography / age-band / rate_scope / unit, and — the strongest design choice — bite refuses to stage without its denominator. All tested.
  • Identitylpc:jobs DISTINCT from dwp_hbai:persons and uk_hmrc:individuals; rate_scope keeps adult-rate vs age-band vs regional claims from colliding.
  • held_out — all 119 → HELD_OUT with an ASHE-jobs-vs-FRS-survey basis; never_calibrate permanent-holdout precedence is checked before the lpc branch.
  • Determinism/mirror — data/lanes.json byte-identical to the app/public mirror; the only rebuild "drift" was CRLF-vs-LF from running under Windows Python (CI produces LF), an environment artifact, not a lane defect.

One cosmetic nit: a uk_aliases comment says the region is "aliased here, once," but the "East of England"→"East" alias actually lives in the adapter REGIONS map (functionally correct — staged output never contains "East of England"; the comment is just slightly misleading). No partner test files touched. Approving.

The uk_aliases comment said the East-of-England alias lives 'here,
once'; it actually lives in the harvest side's REGIONS map, so the
staged rows never carry the LPC spelling and this registry only ever
ACCEPTS the repo's own region names. Functionally identical, but the
comment pointed at the wrong file.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
@vahid-ahmadi

Copy link
Copy Markdown
Contributor Author

Thank you — and the nit is a fair catch, fixed in the latest push.

You're right that the alias doesn't live where the comment said. "East of England" → "East" is applied on the harvest side, in the adapter's REGIONS map, so the staged rows never carry the LPC spelling and uk_aliases only ever accepts the repo's own region names. The comment now says that instead of claiming the alias is registered there.

On your CRLF observation — that's the same thing David hit on #92, and there's now a .gitattributes on that branch forcing -text for the vendored .csv/.csv-metadata.json/.jsonl artifacts. It doesn't cover .xlsx because those are binary and git won't touch them, so this lane's pins are safe either way; but if the LPC lane ever grows a text-shaped artifact it should inherit that rule.

Noted on the rest — particularly that you spot-checked the UK 2025 cells (2,020,000 / 6.6) staging verbatim. That pair is the one I'd most want independently eyeballed, since it's where a count-vs-rate confusion would be invisible.

vahid-ahmadi pushed a commit that referenced this pull request Aug 25, 2026
Found by re-checking every branch for the defect review caught on #92:
DISTINCT is a plain frozenset of "source:value" strings, so nothing stops
it naming a source with no registered vocabulary. A pair like that reads
as if a distinction had been checked when there is nothing on the other
side to check against — and because the pair is just a string, every
test asserting it exists still passes. Mine did.

This branch claimed DWP's age bands DISTINCT from LPC's minimum-wage age
bands. The distinction is real, but the `lpc` vocabulary is registered on
#93's branch and not here, so on this branch the assertion pointed at
nothing. Deferred with a comment naming the prerequisite, and stated in
prose in source.json where it is a sentence rather than a ledger entry.

Two occurrences is a class, so it gets a guard rather than a third
correction. tests/test_uk_aliases_ledger.py reads the registered sources
out of uk_aliases itself — no hardcoded list — and refuses:

  - a DISTINCT pair naming a source with no registered vocabulary
  - the stricter form: a pair naming a VALUE that is registered nowhere,
    since "ukmod:q9" would be equally unfalsifiable

plus a check that the guard is not passing trivially on a near-empty
ledger. Verified by reintroducing the dangling pair and watching it fail.

Suite 283 passed, two builds agree on content_hash, ruff format clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
vahid-ahmadi pushed a commit that referenced this pull request Aug 25, 2026
source.json cited the LPC lane as "(#93/PR #93)", conflating the issue
with the pull request. The issue is #88; #93 is its PR. The uk_aliases
comment and the test both say "#93's branch", which is correct — it is
the branch that carries the vocabulary — so only the source.json note
needed correcting.

Small, but this note exists to tell a future reader where the other half
of a deferred assertion lives, and a wrong pointer defeats the point of
writing it down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Minimum wage coverage and bite (mode 1): PE-UK models the NLW/NMW and nothing validates it

2 participants