[anth-volk]

Fixes #4087

Summary

The Public API now sends stored households to the Simulation Entrypoint's v2 household functions, records which catalog versions offer SPM settings, and checks both against its own /calculate in staging. Nothing calls the new routes yet: no frontend traffic reaches them.

• POST /v2/household-calculations takes country_id, household_id, an optional policy_id, optional axes and add_missing_variables. It needs no account.
  • It checks that the country's countries row allows household calculations (422 naming the country otherwise; a country with household calculations only is accepted), that the household and policy exist in that country, through the same services as /v2/households and /v2/policies (404 otherwise), and that the household has a default year (422 otherwise).
  • It selects the country's running catalog version and sends the Simulation Entrypoint only IDs: the country, the household's and policy's IDs, the options (axes as given) and that version's PolicyEngine.py version. The Simulation Entrypoint reads the household, the policy's parameter values and the catalog version from these tables itself, through its own SELECT grant, converts the household, and calculates the household's default_year, which it sends the household function as the request's required year: the year of added variables, and of values and axes without a period.
  • The answer is 202 with the calculation's ID and status. Callers cannot force a new calculation: an identical one that succeeded in the last 28 days is returned again.
• GET /v2/household-calculations/{calculation_id} returns the calculation as it stands: once it succeeded, the calculated household with its warnings, the PolicyEngine.py version and, for US households, spm_config and spm_provenance; once it failed, its error and the household function's error detail.
• Simulation service answers. The submission and each answer are checked against the generated simulation client's models, and kept in their own key order, because the household function lists warnings in the household's key order. Axes are checked against the service's limits (equal point counts within a group, at most 10,000 points) before anything is sent.
  • 422, passed through with the simulation service's reason: the caller's data it refuses, which it checks before writing anything: a policy changing a parameter the catalog version lacks, a stored policy value it cannot take (such as an object), or a stored household it cannot take (such as a group no person belongs to, which /v2/households accepts). A request field it refuses gets 422 with each field.
  • 503: its other refusals (a household, policy or catalog version it cannot find, or a household without a default year, which the Public API checks first, so such an answer means the two services disagree; or no v2 worker for the version), an unreachable service, an answer of another shape (including a 422 whose body is not text or a list of fields), a gateway token that cannot be fetched, or missing configuration.
• SPM availability in the catalog. Catalog publication records, for each US catalog version, whether the Simulation Entrypoint's public GET /versions lists a certified SPM capability for its PolicyEngine.py version, with the settings schema and certified defaults; /v2/model-catalog serves it as spm. A version the service does not serve yet, or an unreadable answer, leaves it unrecorded for the next publication run. /versions itself is unchanged.
• Route group and logs. /v2/household-calculations belongs to the household route group. Its migration request log records the Supabase read and the v2 workers on the Cloud Run Simulation Entrypoint, not the household group's v1 database and compute flags. The contract registry and the generated migration contract documents include both routes.
• Staging checks. The parallel live staging job runs, against each tagged staging candidate: a submit, poll and result cycle; six calculations submitted at once, each accepted within 10 seconds and finished within 300 seconds; a household across 401 axis points, which must finish; and a parity set of three US and three UK households (with axes, reforms stored through /v2/policies, added variables, and a deprecated input that produces a warning), compared value by value with the Public API's /calculate and /calculate-full. The checks cannot see how many Simulation Entrypoint instances served them or how much memory the household functions used. The job's time limit rises from 20 to 30 minutes.

Route groups covered: household (new routes only). Everything else stays on its current path: /calculate, /calculate-full and the stored-household calculation routes still calculate in process. User-visible API changes: the two new routes, and spm in /v2/model-catalog.

The parity household that adds missing variables names its person you, because /calculate-full adds them for the first age period of a person named you (or else the current year) while the v2 path adds them for the household's default year; a test with the date fixed in 2027 checks both use the same year.

Testing

Run locally on the branch head (with the simulation client generated from PolicyEngine/policyengine-sim-api#957 at 3a2bbe02 installed over the locked release):

• uv run pytest tests/unit tests/contract: 3717 passed;
• against a disposable PostgreSQL 17 container: tests/integration/test_alembic_v2_lifecycle.py and tests/integration/test_v2_catalog_spm_availability.py, 14 passed;
• ruff 0.15.12 (format --check and check), mypy, and scripts/run_quality_guards.py pass; scripts/export_migration_contracts.py regenerated the contract documents;
• the two new live modules ran against a local server standing in for a staging candidate (stored households and policies, calculations, /calculate and /calculate-full, a 503 on each first submission): 9 passed; a lasting 500 fails at once and a lasting 503 fails after the outage limit. They have not run against staging: they run only in the staging deployment job.

Environment variables and operator steps

Operator step: a SELECT grant. The Simulation Entrypoint's runtime role simulation_runtime needs SELECT on tax_benefit_models, tax_benefit_model_versions, regions, datasets, parameters, parameter_values, policies and households, granted by the owner of these tables. Until then, calculations answer 503. It stays an operator step for now: once that role exists, a later change declares the grant in the v2 privileges manifest (policyengine_api/data/v2/privileges.py), so each deploy grants and checks it. This PR does not add it there, because the manifest's deploy-time reconciliation would fail in an environment where the role does not exist yet.

• SIMULATION_ENTRYPOINT_URL
  • Provider: GitHub environment secret, set in staging and production.
  • Runtime: already in the job-level env: of both Cloud Run deploy jobs, and written to the revision by deploy_cloud_run_candidate.sh when it is set. The new routes always call this URL, whatever SIM_ENTRYPOINT selects (both environments select cloud_run_simulation_entrypoint). Without it, both routes answer 503.
  • Catalog publication (new): passed in the publish step's env: in seed-v2-database.yml; that job declares environment: ${{ inputs.deployment_environment }} and is called with secrets: inherit. Publication stops before extraction when it is unset.
• GATEWAY_AUTH_ISSUER, GATEWAY_AUTH_AUDIENCE, GATEWAY_AUTH_CLIENT_ID, GATEWAY_AUTH_CLIENT_SECRET_RESOURCE: organization secrets, unchanged. The deploy step passes them to the deploy script, which writes them with GATEWAY_AUTH_REQUIRED=1; the client secret is read from Secret Manager at runtime, as for every existing simulation call.
• Live checks: API_BASE_URL and STAGING_API_TEST_PROBE_ID come from the deploy job's outputs in the live job's step env:, as before. STAGING_HOUSEHOLD_PARALLEL_COUNT, STAGING_HOUSEHOLD_SUBMIT_LIMIT_SECONDS, STAGING_HOUSEHOLD_FINISH_LIMIT_SECONDS, STAGING_HOUSEHOLD_CALCULATION_TIMEOUT_SECONDS and STAGING_HOUSEHOLD_TRANSIENT_LIMIT_SECONDS are optional limits with defaults; none is set.

Not verified: that the SELECT grant exists in staging or production; that each environment's SIMULATION_ENTRYPOINT_URL names that environment's Simulation Entrypoint (the existing release check covers this); and that the gateway's Auth0 client holds the permissions the Simulation Entrypoint will require for its household routes once it enforces them.

Waits for

• Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042, the base, and the pull requests below it: Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023, Publish v2 catalog regions from checked-in reference files and the certified data release #4020, Rename the v2 table user_policies to user_policy_associations #4010 and Let saved reforms and households point at a new v2 policy or household #4005.
• Add a v2 countries table that matches one country list in code, GET /v2/countries and a startup check #4012 (the countries table) and Check each release calls its own environment's Simulation Entrypoint #4003 (the release check), merged into this branch with plain merge commits. The countries revision is re-parented onto the catalog revisions in this branch only.
• Calculate and store households through the v2 household functions policyengine-sim-api#957 (the household calculation routes), merged and deployed to staging first, with the SELECT grant above. The new staging checks call those routes, staging promotion waits on them, and production waits on staging promotion, so merging this earlier stops every release.
• A release of policyengine-api-simulation-client that contains those routes (an operator step), then raising this PR's dependency (>=0.20261027.288, the latest published release, which lacks them) and the lock to it. Until then, the unit tests that import the client adapter fail in CI, and the routes would answer 500.
• The v2 Alembic revision aeb5f3e8f662 (SPM availability) follows fb1a3e40d3d6 (countries). Migration pull requests in the v2 chain merge one at a time; another open revision on the same parent is moved onto the new head by whichever merges second.

Suppressions added

• # noqa: PLC0415 twice in fastapi_routes/dependencies.py: the household calculation service and its simulation client are imported inside the default factory, so the generated client is loaded only on the first calculation request, as the other default factories do.
• # type: ignore[import-untyped] for policyengine_observability in the simulation client module: the package has no type information, and the module is in mypy's file list.

CI: the Pull request workflow on 4c515e2 failed three jobs, as on every run since the branch was opened.

• Type-check API v2 and Test: the locked policyengine-api-simulation-client 0.20261027.288 lacks the household calculation models, so mypy cannot find them and tests/unit/v2/test_household_calculation_simulation_client.py fails at import. This is the client release this PR waits for.
• Cross-database integration tests: test_v2_catalog_installed.py::test_installed_policyengine_catalog_is_complete_and_bounded and the eight test_v2_catalog_metadata_parity.py tests fail with "the installed PolicyEngine.py does not publish catalog format 2's fields". Those tests come from Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 in the base; the base PRs' runs skipped this job, and this PR changes neither the catalog extraction nor the installed PolicyEngine.py.

Every other job passed, including lint, the quality guards, both Alembic qualifications and the container builds.

🤖 Generated with Claude Code