[anth-volk]

Fixes #4022

Important

This PR must not merge before its PolicyEngine.py bump commit. The Public API pins PolicyEngine.py 6.10.0, which does not publish catalog format 2's fields. On it, every catalog extraction stops with "the installed PolicyEngine.py does not publish catalog format 2's fields", the seed job's catalog publication during a deploy included. PolicyEngine.py 6.11.0 does not contain the fields either. The bump commit waits for:

1. a PolicyEngine.py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701;
2. the simulation API supporting that bundle (the "Ensure PolicyEngine bundle is supported by simulation API" check).

The PRs stacked above this one wait with it.

Summary

Catalog format 2 stores the metadata the app's v3 workspace reads, taken only from PolicyEngine.py's public records (PolicyEngine/policyengine.py#701), never from a country package's tax-benefit system or a private PolicyEngine.py method. This branch is rebased onto the final #4020, which contains master (now pinning PolicyEngine.py 6.10.0).

• Schema. Revision f5c1d9b4685b (on fa7fa1035444) was autogenerated against the v2 configuration with no corrections. It adds nullable columns with no defaults, so catalog versions published earlier keep them empty, and it writes no data:

  • tax_benefit_model_versions: catalog_format, entities, basic_inputs;
  • variables: is_input_variable, hidden_input, module_name, adds_parameter_path, subtracts_parameter_path, enum_values, unit, definition_period;
  • parameters: economy, household.

  entities and enum_values are typed with CatalogEntity and CatalogEnumValue.

• Extraction.

  • normalize_model_metadata reads entities (key, plural, person flag), basic inputs, variable flags and modules, the parameter path that adds or subtracts names (beside the list PolicyEngine.py resolves, which stays), enum values with their declared labels (an integer label stays an integer), each variable's unit and definition period, and parameter flags.
  • The unit (for example currency-USD) and the definition period (year, month or eternity) are needed by the app's household builder and breakdown on v2: currency formatting, "per month" and "per year" labels, and household validation. PolicyEngine.py's public Variable record carries both (6.7.5 and 6.10.0 already do).
  • Bracket schedules arrive as public parameter nodes, with their stored label or none.
  • Extraction first checks that PolicyEngine.py publishes these fields and names each one missing. It then fails with a specific message on:
    • other than one person entity;
    • a basic input or parameter path that is not published;
    • enum labels that do not match the possible values or are not JSON scalars;
    • a flag that is not a boolean.
  • An empty basic-input list is stored as empty.

• Default datasets come from PolicyEngine.py. By the user's decision, each country's default dataset is the default_dataset the installed PolicyEngine.py release names in its manifest. This PR removes the Public API's own list (REVIEWED_DEFAULT_DATASETS). Publication still stops if the manifest names no default, if the release does not certify it, or if the release routes national regions to another dataset.

• Storage and comparison. Publication stages and inserts the new fields. When the same PolicyEngine.py version is published again, a version stored before format 2 (null catalog_format) is compared without the format 2 fields and keeps them empty; a format 2 version must match on them too. Publication never rewrites a stored version.

• Parity check. tests/integration/test_v2_catalog_metadata_parity.py (opt-in through RUN_V2_CATALOG_COMPATIBILITY=1) compares every new field with /{country}/metadata for the US and the UK. Enum labels are compared with their types. The v2 integration check runs it in its existing installed-catalog step.

Parity with /{country}/metadata

Measured with PolicyEngine.py #701's branch installed locally (6.10.0 with the new fields). 0 mismatches in every field:

Field	US	UK
Entities (key, plural, person flag, in order)	6	3
Basic inputs	3	5
Variables compared	6,167	1,170
Enum variables (labels and their types)	290	37
Input variables	925	300
Hidden inputs	54	0
Units (variables with one; 15 distinct in each country)	3,955	708
Definition periods (year / month / eternity)	4,210 / 1,953 / 4	1,160 / 10 / 0
Parameter-named adds	237	4
Parameter-named subtracts	2	0
gov parameters (economy and household flags)	100,276	1,551

Every gov bracket schedule is a catalog node, and its stored label matches the label /{country}/metadata shows when one is stored.

With #701's branch installed, the full installed extraction succeeds:

• US: default populace_us_2024; 51 states and 436 congressional districts as row filters on populace_us_2024_acs_local. Region data verification against the real files passes.
• UK: default microcosm_uk_2024_25; the national region and four nations on it; 650 constituencies and 361 local authorities as row filters on microcosm_uk_2024_25_dense. Region data verification of the UK files could not run locally, for lack of read access to the UK data, so it first runs in the seed job.

What the PolicyEngine.py bump commit must change

• policyengine[models]==6.10.0 in pyproject.toml and docker/Dockerfile → the release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701, then uv lock.
• The changelog fragment the bundle update script writes, changelog.d/update-policyengine-bundle-<version>.changed.md ("Update the PolicyEngine bundle to .").
• tests/integration/test_v2_catalog_installed.py: the parameter_nodes count, from 29,307 to 30,214 (the bracket schedules Update PolicyEngine US to 0.434.0 #701 publishes as nodes), and the comment above the counts that names the 6.10.0 bundle. With Update PolicyEngine US to 0.434.0 #701's branch installed on 6.10.0's country packages, every other count and assertion in that test already holds.
• Nothing in tests/integration/test_v2_catalog_publication_qualification.py. It also calls extract_installed_catalog, so it fails on the locked 6.10.0 too, and it passes with Update PolicyEngine US to 0.434.0 #701's branch installed (measurement below).
• Nothing about default datasets: they now come from the release.

Production-scale publication with format 2 fields

RUN_V2_CATALOG_PUBLICATION_QUALIFICATION=1 pytest tests/integration/test_v2_catalog_publication_qualification.py, with #701's branch installed and a disposable Postgres 17, passed:

• Peak additional publisher memory: 8,147,310 bytes (7.8 MiB), against the test's 256 MiB limit (8,146,245 bytes before the unit and definition period were added).
• Publication time: 53.9 s for the whole catalog on a shared machine (86.5 s in an earlier, more loaded run).
• Rows published: 7,337 variables, 30,214 parameter nodes, 104,172 parameters, 1,194,360 parameter values, 4 datasets and 1,504 regions. 333 US places are left out.

Migration notes

• Route groups: none change. This prepares stored metadata that a later PR serves through the v2 catalog routes.
• API v1 routes: /{country}/metadata and the other v1 routes are unchanged.
• User-visible contract changes: none.
• Migration: one generated Alembic revision on the v2 chain. It merges one at a time with any other open v2 revision.

Environment variables

None read or added. The v2 integration workflow only adds a test file to an existing step, whose RUN_V2_CATALOG_COMPATIBILITY stays a literal "1".

Testing

GitHub Actions runs on this branch are cancelled at the user's request, so the evidence is local. Commits are listed from the bottom of the branch.

Review. Two review rounds. The first found no substantiated bugs, and its quality findings are fixed. The second reviewed the default-dataset change and also found no substantiated bugs; its findings on this description and the documentation are fixed. A third review, of the unit and definition period, found no substantiated bugs; its quality findings (test names, and limits written twice) are fixed.

Commit	Failing run first	Passing run
d66267943 columns and revision	column test KeyError: 'catalog_format'; revision chain and new-revision tests (no child of fa7fa1035444); lifecycle test KeyError with the revision removed	focused unit tests; v2 lifecycle tests; alembic check clean
14673dc46 extraction and parity test	new modules failed at import (CATALOG_FORMAT, normalize_model_metadata); workflow test did not find the parity test	21 unit tests; parity 8 of 8 on the US and UK
07b7b0cfe storage and comparison	staging rows without the new columns; stored fields all null; changed format 2 field "DID NOT RAISE"	publication, route and lifecycle integration tests
b8a75d12c naming missing fields	8 cases failed; on a release without the fields the error was "variable age_group enum values do not match its possible values"	79 focused tests
a94b2e2d4 label types in the parity test	with integer labels turned into floats on purpose, the old comparison passed and the new one found 1 mismatch	parity 8 of 8
cca58e31a round-trip and change detection	with the fields left out of the comparison and labels stored as strings on purpose, all 4 new tests failed	integration and catalog unit tests
e46611fca typed JSON columns	ImportError: CatalogEntity	models, unit and integration tests
494f43465 default datasets from the release	5 failed against the previous code: a release default other than the fixed list was refused ("does not match the reviewed selection"), and the missing, empty, uncertified and national-route cases gave the old message	62 extraction tests; 188 catalog, Alembic and model unit tests
1f69e9144 documentation: 6.10.0 also lacks the fields	documentation only	67 extraction and metadata tests; ruff, mypy and quality guards
5c565e61f unit and definition period columns; revision f5c1d9b4685b regenerated with autogenerate (--rev-id f5c1d9b4685b, parent fa7fa1035444), now 13 columns	column test KeyError: 'unit'; revision test assert 11 == 13; lifecycle test on the previous revision KeyError: 'unit'	75 model, revision and lifecycle tests; alembic check clean; 1,129 v2 unit tests
376846411 extracting the unit and definition period	5 unit tests failed (record without unit, missing-field and type messages); parity test AttributeError: 'VariableRecord' object has no attribute 'unit'	31 unit tests; parity 8 of 8; 1,132 v2 unit tests
7c0eac46f storing and comparing them	staging rows without the two columns; comparison column counts [3, 6, …]; stored values missing; a changed unit or period "DID NOT RAISE"	11 staging and publication unit tests; 40 publication, route and lifecycle integration tests
7e0fb9518 test and fixture names that name the unit and period	tests only	the renamed tests pass
f803106ee text limits from the column lengths	ImportError: VARIABLE_TEXT_LIMITS; the over-long cases passed before too, as the limits already existed	100 extraction tests, including one over-long value per format 2 text field; 1,138 v2 unit tests

On the final branch, under the locked PolicyEngine.py 6.10.0:

• the full make test set: 3,521 passed;
• publication, route and lifecycle integration tests on a schema recreated at head on a disposable Postgres 17: 40 passed;
• alembic check: no new operations;
• ruff, ruff format, mypy, scripts/run_quality_guards.py, python -m scripts.rules validate and guard: clean;
• scripts/export_migration_contracts.py: no changes;
• the opt-in tests that call extract_installed_catalog fail with the missing-fields message: test_v2_catalog_installed.py (1 test), test_v2_catalog_publication_qualification.py (1 test) and every parity test (8). That is expected until the bump commit.

With #701's branch installed:

• the full make test set: 3,507 passed (before the final rebase);
• the parity test, including the unit and definition period: 8 passed;
• the publication qualification: passed (measurement above);
• test_v2_catalog_installed.py: differs only in the parameter_nodes count listed above.

There are no inline # noqa suppressions and no changes under rules/.

🤖 Generated with Claude Code