[anth-volk]

Fixes #4019

Summary

Catalog publication now reads each country's regions from region reference files checked into the Public API, instead of copying PolicyEngine.py's region registry. This branch is rebased onto #4010's final commits, which include master, so it targets the PolicyEngine.py release the Public API pins: 6.10.0.

The review of the retarget to 6.10.0 and of Ards and North Down found no bugs. Two small additions followed:

• unit tests now cover a UK local authority's routed dataset, column and value, that the UK records no fallback, and a release that routes no dataset to local authorities;

• the Stage 9 runbook paragraphs are rewrapped.

• Reference files. policyengine_api/data/v2/catalog/region_reference_files/us.csv and uk.csv list every region's code, label, type, parent, US state, the value that selects its households, and the v1 identifier it replaces:

  • country/ni becomes country/northern_ireland;
  • constituency/Aldershot becomes constituency/E14001063;
  • local_authority/Maidstone becomes local_authority/E07000110;
  • local_authority/Ards and North Down becomes local_authority/N09000011.

  US codes are unchanged. The national labels are "United States" and "United Kingdom". A loader checks that codes and v1 identifiers are unique, that each code starts with its type, and that every parent leads to the national region.

• Scoping from the certified data release. One rule per country and region type takes the dataset from the release manifest's region datasets and sets a row filter:

  Regions	Dataset	Filter
  US national	populace_us_2024	none
  US states and congressional districts	the release's state and district dataset (populace_us_2024_acs_local)	row_filter on state_fips or congressional_district_geoid
  UK national	microcosm_uk_2024_25	none
  UK nations	microcosm_uk_2024_25	row_filter on region
  UK constituencies and local authorities	the local-area dataset the release routes them to (microcosm_uk_2024_25_dense)	row_filter on constituency_code_oa or la_code_oa, value the ONS code

  The UK national file has no area codes. A UK constituency or local authority therefore never falls back to it: a release that routes no dataset to their type stops publication, naming the region.

• Verification before publishing. Before publishing a catalog version that is not stored yet, the publication command reads each dataset the filters use from Hugging Face, at its pinned revision, using ranged requests. Only the 1 MB blocks holding HDF5 metadata and column names are fetched. Publication stops if a filter's column is not a household column of its dataset. Errors name only artifacts, tables, columns, codes and counts.

• Omitted regions. Region types without a rule are left out and counted in a new omitted_region_summaries evidence field. Today that is only US places.

• Stored versions keep their regions. See "Stored catalog versions" below.

• Direct dependencies: h5py>=3.16,<4 and huggingface-hub>=1.12,<2. The locked versions are unchanged; uv.lock changes only in the project's own metadata.

No route, table or migration changes. /v2/regions serves whatever the catalog stores.

Retargeted to PolicyEngine.py 6.10.0

Earlier versions of this PR scoped UK constituencies and local authorities with a weight_replacement filter on the release's weight matrices. That path, its weight-matrix checks and its matrix reading are removed, because no release the Public API can publish uses them:

• the pinned 6.10.0 release certifies no weight matrices;
• stored 6.7.5 versions keep their regions and are never scoped again;
• the Public API's local-authority list now has 361 rows while 6.7.5's matrix had 360, so the matrix check would be wrong for 6.7.5 anyway.

Ards and North Down (N09000011)

Included by the user's decision. The Public API's own local-authority list on master already has it, and the v1 metadata now serves local_authority/Ards and North Down. Under 6.10.0 it is row-filtered on la_code_oa in the local-area dataset, so it publishes.

Whether that column holds rows for N09000011 is not checked. That would need reading the column's values from the private UK dataset, so it stays an open item.

Counts

• UK: 1,016 regions (1 national, 4 nations, 650 constituencies, 361 local authorities), all published.
• US: the file lists 821 regions (1 national, 51 states, 436 districts, 333 places). 488 are published; the places are left out.
• Total: the installed 6.10.0 catalog extracts 1,504 regions and 4 datasets.

Sources of codes and labels

• UK constituencies and local authorities come from the Public API's own lists, policyengine_api/data/constituencies_2024.csv and local_authorities_2021.csv. The local-authority list now has 361 rows and matches PolicyEngine.py 6.10.0's 2023 list. A nation is the parent of each, chosen from the first letter of its ONS code (E, S, W, N).
• US states and districts come from the Public API's congressional_districts module. US FIPS numbers and places come from PolicyEngine.py's public data.
• Every US region and UK nation matched PolicyEngine.py 6.7.5's registry in labels, parents, state fields and filter values. The US file is unchanged when regenerated with 6.10.0.

Stored catalog versions

• The rule. A catalog version is identified by its country and PolicyEngine.py version, and never changes once published. For a stored version, publication still compares every other table exactly, but keeps its region rows as they were published.
  • It prints a warning with the country, the version and the counts of differing region rows.
  • Its evidence counts the stored regions, lists the version under kept_regions with stored and staged counts, and leaves out that country's staged fallback and omitted summaries.
  • A difference in any other table still fails, and the error names the table.
• Why. Published versions must not change; 1.16 caches its responses as immutable. The seed job republishes the installed version on every release.
• Trade-off. A stored version's region rows are no longer compared when it is republished. If they were damaged, publication would print the difference instead of failing.

When the new regions reach staging and production

Only when a catalog version that is not yet stored is published. Where the deploy of master has already published the PolicyEngine.py 6.10.0 catalog, that version keeps the regions taken from the registry, because a stored version keeps its regions. The reference-file regions then reach staging and production with the next PolicyEngine.py version that is published, by an operator or by the release workflow's seed job. This PR does not publish anything.

Open risks

• Nothing automated reads the real certified files before the release workflow's seed job publishes the next catalog version. If an artifact's layout differs from what the reader understands, or the read token cannot read the UK data, that seed job stops before publishing.
• Earlier, the US check passed locally against the real populace_us_2024_acs_local file (9.8 GB) in 55.5 seconds.
• The UK datasets could not be read locally: the local token has no access. The reader handles the HDF5 layouts policyengine-uk writes, tested with local fixture files.
• Whether la_code_oa holds rows for every local authority, N09000011 included, is not checked.

Follow-up

US places stay left out until a place filter's value format is checked and compute accepts places. The certified ACS-local file has a place_fips household column.

Environment variables

Variable	Read by	Provider	How it reaches the job
HUGGING_FACE_TOKEN	The catalog publication command, to read certified datasets	The GitHub organization secret PE_UK_PRIVATE_HF_READ_TOKEN, listed as available to this repository, with no environment-level secret of the same name	env: of the catalog publication step only ("Seed and validate the v2 metadata catalog") in seed-v2-database.yml. It arrives through secrets: inherit from the release workflow's staging and production seed jobs, and directly on manual dispatch. A test checks that the workflow names it only once.
V2_DATA_WRITE_DATABASE_URL	Catalog publication	Unchanged	Unchanged

Not verified: that PE_UK_PRIVATE_HF_READ_TOKEN can read the UK data release. There has been no live run. If a UK version needs checking and the token is unset, the command stops before reading anything, with a configuration error naming the variable.

Suppressions

None added.

What this waits for before merging

• Rename the v2 table user_policies to user_policy_associations #4010 and Let saved reforms and households point at a new v2 policy or household #4005 below it in the stack.
• Review.
• The read token's access to the UK data release, confirmed in a live run or by an operator.

Testing

GitHub Actions runs on this branch are cancelled at the user's request. The evidence is these local runs on the tip:

• Full make test set: 3476 passed.
• v2 integration tests on a disposable Postgres:
  • catalog publication and metadata routes: passed;
  • installed catalog against PolicyEngine.py 6.10.0 with RUN_V2_CATALOG_COMPATIBILITY=1: passed;
  • production-scale publication qualification: passed, publishing 1,504 regions.
• ruff format --check, ruff check, mypy on the configured set, python scripts/run_quality_guards.py, python -m scripts.rules validate and guard, alembic -c alembic-v2.ini check: clean.
• Changelog fragment: changelog.d/v2plan-1f-123-region-reference-files.changed.md.

🤖 Generated with Claude Code