Skip to content

Refuse new policies whose data names paths that are not parameters - #4123

Merged
anth-volk merged 4 commits into
masterfrom
policy-path-validation
Oct 30, 2026
Merged

anth-volk merged 4 commits into
masterfrom
policy-path-validation

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4122

Summary

  • Policy create checks new policies. POST /<country_id>/policy is the only route that creates a v1 policy. When a request would store a new policy (the 201 case), PolicyService.set_policy now checks every key in data against PolicyEngineCountry.parameter_names (added by Report which parameters a policy sets that the current model no longer has #4039): the names of every Parameter in the country's loaded tax-benefit system, the same system the Public API uses for its metadata and household calculations. The route reads them through Report which parameters a policy sets that the current model no longer has #4039's loaded_model_parameter_names, and the comparison is parameters_missing_from_model, the one the v2 copy and missing_parameters use. Scale bracket values such as gov.irs.credits.ctc.amount.base[0].amount and contributed parameters the model defines are accepted. Unknown paths, parameter groups, scales and scale brackets get a 400 whose message and errors list every refused path, sorted. Nothing is stored.
  • Re-saving is unchanged. The check runs only after the lookup for an identical stored policy (same country, content hash and label) finds none, so re-posting a stored policy that uses since-removed parameters still returns it with a 200.
  • Household check refuses groups. find_unrecognized_policy_inputs tested only whether a policy key appears in the metadata's parameters, which also lists every group, scale and bracket under gov as "type": "parameterNode". A group therefore passed the check, reached the model, and failed with a 500 ('ParameterNode' object has no attribute 'values_list'). The check now accepts only entries of type parameter, so a group gets the existing 400. /metadata is unchanged.
  • This prevents the unknown-path copy failures counted in [telemetry] policyengine-api: logged error on /<country_id>/policy (82fe6ed2c0735fa9) #4098: those policies, such as 98762 (gov.irs.credits.ctc.amount), 98768 (gov.contrib.states.mt.hb1.in_effect) and 98770 (gov.x), can no longer be created.
  • The OpenAPI document gains the 400 response for policy creation.
  • No noqa suppressions or type: ignore comments are added.

Example 400:

{
  "status": "error",
  "message": "Policy data contains keys that are not parameters in policyengine-us 2.2.1: gov.contrib.states.mt.hb1.in_effect, gov.irs.credits.ctc.amount, gov.x",
  "result": null,
  "errors": [
    {
      "type": "policy_parameter",
      "name": "gov.contrib.states.mt.hb1.in_effect",
      "path": "data.gov.contrib.states.mt.hb1.in_effect",
      "message": "Unrecognized policy parameter `gov.contrib.states.mt.hb1.in_effect` at `data.gov.contrib.states.mt.hb1.in_effect`."
    }
  ]
}

(errors holds one entry per refused path; the other two are omitted here.)

Testing

  • Tests added first, in their own commit, then the fix. Against the code before the fix, 22 of the 32 new tests fail (19 assertions and 3 that cannot import the new error type); the other 10 cover valid paths, re-saving, and household cases that already returned 400.
  • New tests cover gov.x, a parameter group, scale and scale bracket, a missing gov.contrib path, and valid scale bracket and contributed paths, for both the US and the UK, through the policy create route and the household calculate route, plus service tests with an injected parameter catalog.
  • Focused run of the new and affected suites: 197 passed.
  • Before the rebase onto master (with Report which parameters a policy sets that the current model no longer has #4039 and Return 400 for household axes the simulation cannot expand #4119), make test passed 3909 tests with 0 failures. After the rebase, the test files this pull request adds or changes pass: 146 passed.
  • python scripts/run_quality_guards.py: 5 of 5 guards passed. python scripts/export_migration_contracts.py left the generated files unchanged.
  • ruff format --check . (746 files already formatted), ruff check ., and mypy (169 source files, no issues) pass. The changelog fragment check finds changelog.d/4122.fixed.md.
  • tests/unit/routes/test_policy_dual_write_routes.py and tests/to_refactor/python/test_policy.py created policies with placeholder paths through the real service; they now use real US parameters. Test metadata fixtures whose parameter entries had no type now give them "type": "parameter", as the real metadata does.
  • The live staging reform payloads in tests/data (Utah, California EITC, Oregon, UK Universal Credit, and the economy test policy) set only settable parameters, so the deployed checks keep working.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.75000% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.15%. Comparing base (e615bf3) to head (af1eba9).
⚠️ Report is 31 commits behind head on master.

Files with missing lines Patch % Lines
policyengine_api/services/policy_service.py 91.30% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4123      +/-   ##
==========================================
+ Coverage   88.13%   88.15%   +0.02%     
==========================================
  Files         273      274       +1     
  Lines       16299    16393      +94     
  Branches     2668     2689      +21     
==========================================
+ Hits        14365    14452      +87     
- Misses       1268     1271       +3     
- Partials      666      670       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 3 commits October 30, 2026 00:10
A new v1 policy may name only parameters that the running country model
can set a value on. These tests cover unknown paths, parameter groups,
scales, scale brackets and missing contributed parameters for the US and
the UK, plus valid scale bracket and contributed parameters, and check
that re-saving an identical stored policy still returns it.

They also cover the household calculation check, which currently accepts
parameter groups because the metadata lists them as parameterNode
entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /<country_id>/policy now checks every key in data when the request
would store a new policy. Each key must name a Parameter in the country's
loaded tax-benefit system, the one the Public API uses for its metadata
and household calculations, so scale bracket values and contributed
parameters the model defines are accepted. Unknown paths, parameter
groups, scales and scale brackets get a 400 whose message and errors list
every refused path. Re-saving a policy identical to a stored one still
returns it with a 200, without checking its paths.

The household calculation check tested only whether a policy key appears
in the metadata's parameters, which also lists groups, scales and
brackets as parameterNode entries. It now accepts only entries of type
parameter, so a group gets the existing 400 instead of reaching the
model and failing with a 500. The metadata is unchanged.

Fixes #4122

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test created a policy that sets sample_parameter, which the US model
does not define, so policy creation now refuses it with a 400. It sets
gov.contrib.states.mt.ctc.in_effect instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UnsettableParameterPathsError now returns the 400 response's errors
entries itself, so the policy route passes them through instead of
converting each one.

A model parameter lookup that returns None, for a country without a
loaded model, now leaves a new policy no parameters to set, so every
path is refused. Every country the policy service accepts has a loaded
model, so this replaces a branch that could not be reached in
production, and a service test covers it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review October 30, 2026 13:13
@anth-volk
anth-volk merged commit 6c4d53a into master Oct 30, 2026
15 checks passed
@anth-volk
anth-volk deleted the policy-path-validation branch October 30, 2026 13:14

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refuse new v1 policies whose data names paths that are not parameters

1 participant