[anth-volk]

Fixes #4048

Summary

Plan item 1.18: one compact parameter search file per catalog version, served at /v2/parameters/search-file, for the parameter search of the app's newer workspace (v3, app/src/libs/parameterSearch.ts). It is stacked on #4046 (1.17), and v3 does not use it until it moves to the v2 routes.

• The file (format 1). One JSON document per catalog version: {format, country_id, policyengine_version, parameters, other_parameters, nodes}.
  • parameters: every parameter a reform can change, by 1.17's shared rule (parameter_tree.is_reform_changeable). Each entry has path, label, unit, description and editable, and, only where it applies, editable_changes.
  • editable says whether a draft can edit the parameter's current-law value: a number, a yes/no value or an infinite threshold, as v3's draft editor allows. Where that changes over time (for example a bracket whose value becomes null from a date), editable describes the earliest value and editable_changes lists {start_date, editable} for each date from which it changes. The file therefore does not depend on the day it was built.
  • other_parameters: the path and label of every other parameter under gov. v3's search reads these for its concept clusters, acronym aliases and member labels.
  • nodes: the path and label of gov and of every node under it that has a stored label, for breadcrumbs, schedule names and concept clusters.
  • Labels are the stored labels, or null. v3's search treats a label that only restates its path's last key as a placeholder, so its reader applies v1's fallback (the key with underscores as spaces, "bracket N" for a bracket).
• Storage. The new parameter_search_files table holds one row per catalog version and file format (unique on both). Each row holds the gzip-compressed JSON (level 9, no timestamp, so the bytes are reproducible), the SHA-256 of the uncompressed JSON, and both sizes. Rows are deleted with their version.
• Revision c614b6bcccfc is chained on 3869eb814a90 (1.17). It is autogenerated with no corrections, creates only the table and writes no rows.
• Writing at publication.
  • Catalog publication writes, in its transaction and under its lock, the file of each version it publishes: each new version, and the installed version when an earlier release stored it before search files existed.
  • The file is built from the installed packages' catalog records, never from rows an earlier release stored. It is a row-only, idempotent insert of package-derived rows, which the Alembic skill's deployment-seeder passage allows, as 1.17's trees are.
  • Writing only inserts. A stored file of the same format must equal the one built from the installed packages, and publication fails if it differs or if a version it publishes is left without a file. A later file format adds a row instead of changing the stored one.
  • Versions stored before catalog format 2 (catalog_format null) get no file. Their parameter rows store no economy or household flags, so a file built from the installed catalog's flags would contradict them. They are exempt from the check above. The stored version's ID and format come from 1.17's shared publication_reconciliation.stored_version, so both writers apply this rule the same way.
  • The evidence adds parameter_search_files_written (model, version, uncompressed and compressed bytes).
• The route. GET /v2/parameters/search-file?country_id=…&policyengine_version=… (1.16's strict query model; the version is optional).
  • The body is the file itself, not the status/result envelope.
  • A client that accepts gzip receives the stored bytes unchanged with Content-Encoding: gzip. Any other client receives the uncompressed JSON.
  • Each representation has its own strong ETag, made from the SHA-256 of the uncompressed JSON: "<sha256>", or "<sha256>-gzip" for the gzip one. A 304 needs the tag of the representation being requested, so a client that switches encoding gets a 200.
  • The route runs under 1.16's CacheableMetadataRoute, so Cache-Control (a day for a named version, five minutes for the running one), Vary: Origin, Accept-Encoding, the CORS exposure of ETag and the 304 are those of every metadata route.
  • A request with If-None-Match first reads only the stored hash. When it names the current tag, the route answers 304 without reading the file.
  • A version without a file of the current format returns 503 with a message that names it and says which versions have files. The 503 is not cached. An unknown version returns 404.

Route contract changes

• New route: /v2/parameters/search-file, with its OpenAPI responses (200 with ETag, Cache-Control, Vary and Content-Encoding headers; 304; 404, 405, 422, 500 and 503). The contract registry gains one request (93 in all).

• Every route: gzip follows Accept-Encoding's qualities. Starlette's GZipMiddleware compressed whenever the text "gzip" appeared anywhere in Accept-Encoding. The new content_encoding.NegotiatedGZipMiddleware replaces it application-wide, with the same 1,000-byte threshold and level 4, and decides with accepts_gzip: an explicit gzip or x-gzip entry decides; otherwise * does; a quality of 0 refuses. On every route:

  • a client that refuses gzip (gzip;q=0, br, gzip;q=0.0) now receives an uncompressed response, where it used to receive gzip;
  • Accept-Encoding: *, br, * and differently cased values such as GZIP now receive gzip, where they used to receive an uncompressed response;
  • a value that only contains the text, such as gzipped, no longer receives gzip.

  The search-file route uses the same rule, so its chosen encoding is final and the middleware never re-encodes it. The middleware subclasses Starlette's GZipResponder and IdentityResponder, which Starlette does not document; the locked Starlette is 1.0.0, and a unit test fails with a clear message if either class or its constructor signature changes.

Measured

Locally, with the .py branch containing PolicyEngine/policyengine.py#701 (6.10.0) installed:

Measure	US	UK
parameters (a reform can change them)	92,707	1,540
of which with editable_changes	4	25
other_parameters	7,569	11
nodes	1,960	161
Compressed (stored, and sent as gzip)	868,406 bytes (0.87 MB)	50,736 bytes
Uncompressed	19,941,379 bytes (19.9 MB)	391,059 bytes
Building and encoding the file from the installed catalog	2.6 s, 113.9 MiB peak traced memory	0.04 s

• Route (the US file in local Postgres, FastAPI test client, 15 requests each): gzip 200 in 24.2 ms median (the first request 118.9 ms); 304 revalidation in 9.7 ms median; uncompressed 200 in 25.2 ms median.

• Publication. The production-scale publication test passes with this PR: 52.6 s, and 115.1 MiB (120,693,440 bytes) of additional publisher memory against its 256 MiB limit. Its evidence lists both files with the sizes above.

• Against the plan's estimate. The plan estimated 0.72 MB compressed for the US (92,706 parameters and 29,310 node labels, measured from v1's metadata on 2026-10-21). The file is 0.87 MB compressed and 19.9 MB uncompressed (the plan's measurement: 12 MB). Without other_parameters it is 0.77 MB compressed (774,574 bytes) and 19.0 MB uncompressed.

• other_parameters is kept: please confirm. v3's search reads labels of parameters a reform cannot change, for its concept clusters, acronym aliases and member labels, so leaving them out could change search results. If they are not needed, removing them saves 93,832 bytes compressed (0.09 MB).

Staging times need a deploy.

v3's search on this file

v3's own search code (parameterSearch.ts, unchanged) was run over its evaluation cases on three inputs, each a v1-shaped snapshot:

• v1's /{country}/metadata as today (the reference);
• a snapshot rebuilt from this file with every label replaced by v1's label for the same path, so only the file's content differs ("v1 labels");
• a snapshot rebuilt from the file as served, with v1's fallback applied where a label is null ("catalog labels").

Cases	Labels	Found in the top 20: today → file	In the top 5: today → file	Rank changes
US tracker (75)	v1	31 → 31	29 → 29	four cases move one place: ga-hb1001 3 → 2, ga-sb476 4 → 3, ok-sb2156 4 → 3, ut-hb210 2 → 3
US tracker (75)	catalog	31 → 32	29 → 29	ga-sb476, ok-sb2156 and ut-hb210 as with v1 labels; also mo-sb458 5 → 1, ms-sb2869 not found → 11, ny-s9077 16 → 18
US programs (140)	v1	128 → 128	127 → 127	none
US programs (140)	catalog	128 → 128	127 → 127	multnomah_pfa:name 3 → 5
UK queries (48)	v1	48 → 48	48 → 48	none
UK queries (48)	catalog	48 → 48	48 → 48	"income tax" 2 → 3, "scottish higher rate" 1 → 2, "scottish intermediate rate" 1 → 2

No case found today is lost, and the evaluation case files are unchanged.

Labels: an open decision for a person

The file carries the catalog's stored labels, which are .py's. .py generates labels for parameters without a stored one (utils/parameter_labels.py): bracket parameters become " (tier N amount)" and breakdown parameters " ()". v1's /{country}/metadata does not have these generated labels: for those parameters it uses the last part of the name with underscores as spaces, and "bracket N" for brackets. The rank changes in the "catalog labels" rows above come from these labels. This is the same question as 1.17's (#4046), and the options are the same:

1. accept .py's labels;
2. shorten generated labels in the search file only;
3. have .py also publish the raw stored label, which changes 1.14 and 1.15.

This PR keeps .py's labels.

Search files for other versions stored earlier: left out

• Publication writes a file only for the versions it publishes, from the installed packages. Writing files for other stored versions would derive rows from data that earlier releases stored, which the Alembic skill's "V2 application-data migrations" section reserves for Alembic and a human migration decision.
• So the installed version gets its file at the first deploy that includes this PR, if it is stored with catalog format 2. Every other stored version returns the 503 until a person approves that migration.

Migration notes

• Route group and runtime path. The metadata route group; the route is served by FastAPI and reads Supabase (no change). Nothing moves off Flask.
• Deploy order.
  • The seed job runs alembic upgrade head, then catalog publication, which writes the files, before the new release takes traffic.
  • The previous release does not read the new table.
  • The revision's downgrade drops the table and keeps every catalog row.
• Privileges. The runtime role may read the table. The catalog writer may read and insert rows. A test publishes as the catalog writer role with only its declared privileges, and another reads the file and its hash as the runtime role.
• Environment variables. None are read or added, and no workflow changed.
• Memory. The production-scale publication test, which allows 256 MiB of additional publisher memory, measured 115.1 MiB with this PR.
• Suppressions. One # noqa: E501 is added, on the import of database_connectors.reads_parameter_search_file in the metadata services module: the dotted module path alone is longer than the line limit and cannot be wrapped. It matches the two imports beside it.
• Rule scope and statement change. rules/registry.toml adds policyengine_api/content_encoding.py to the scope, statement and checked paths of api.quality.mypy-v2, and pyproject.toml adds it to mypy's file list, so the new middleware is type-checked with the v2 code. docs/generated/rules.md is regenerated. A review from PolicyEngine/core-developers is requested.

Follow-up (not changed here)

The other metadata routes give their gzip and uncompressed responses one strong ETag, computed before compression. Strong ETags should differ between encodings. This PR only corrects the shared header description ("a hash of the uncompressed response body, so a gzip-encoded response carries the same tag"). Changing those routes is left for a follow-up.

Waits for

• Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view #4046 (1.17) and the PRs below it: Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042, Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023, Publish v2 catalog regions from checked-in reference files and the certified data release #4020, Rename the v2 table user_policies to user_policy_associations #4010 and Let saved reforms and households point at a new v2 policy or household #4005. Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 (1.15) and the stack above it wait for the .py release containing Publish the model metadata the Public API can only read through a private method policyengine.py#701.
• Review.
• A person's decisions on the labels, on other_parameters, and on the migration for other stored versions (above).

Testing

CI on this branch is cancelled at the user's request. The evidence is local. For each commit that changes tests, its changed test modules were run against its parent's code (failed first) and then at the commit (passed). Postgres modules ran on a database recreated at the checked-out code's head, with the managed roles' privileges applied.

Commit	Failed first	Then passed
016e0b171 Build each catalog version's parameter search file	ModuleNotFoundError for parameter_search_file	11 builder tests
69dfb12ef Store each catalog version's parameter search file in its own table	the revision chain, no child revision of 3869eb814a90, the import side-effect test's module list, KeyError: 'parameter_search_files'; lifecycle: UndefinedTable	77 model, revision and import tests; 11 lifecycle tests; alembic check clean
db9054597 Write the parameter search file of each catalog version being published	evidence field missing; no file rows (NoResultFound); a differing stored file did not raise	9 unit and 33 Postgres publication tests
b40b572c3 Serve a catalog version's parameter search file at /v2/parameters/search-file	ImportError for the content model and the unavailable error; Postgres: no ETag, 422 instead of 503, no service read	193 unit and 25 Postgres route tests
7eb8f5080 Include the labelled gov node in the parameter search file	assert [] == [{'path': 'gov', 'label': 'Government'}]	22 builder tests
6f9ae2ea6 Type the parameter search file and name its editability consistently	ImportError for EditabilityChange	133 tests
32566f316 Describe when parameter search file rows are written in the privilege manifest	comment only	the focused set below
5a1bb4d0a Gzip a response only for a client that accepts gzip	6 tests: gzip;q=0, br, gzip;q=0.0 and gzipped were compressed; br, * and GZIP were not; a large search file sent with gzip;q=0 came back gzip	164 application and route tests
32939ade8 Name the catalog version in the search file route's 503	the message was "The initialized v2 catalog is unavailable."; the OpenAPI 503 named no file case	165 unit and 25 Postgres route tests
4693c89ed Test the search file route's CORS exposure and 404, and its publication roles	tests only; the writer-role test fails with InsufficientPrivilege: permission denied for table parameter_search_files when the grant is revoked and the declaration narrowed	115 unit and 61 Postgres tests
bd986d6af Describe the gzip negotiation fix and the search file's 503 in the changelog	documents only	—
76767f76e Write no search file for an installed version stored before catalog format 2	a file was written for a null-format installed version	36 Postgres publication tests
a5dd07f00 Answer a search file revalidation from its stored hash alone	a matching revalidation read the whole file; no hash read in the service	170 unit and 26 Postgres route tests
c8dd1de00 Declare editable_changes as an optional list that is never null	the shape test, now comparing JSON types, failed on editable_changes	116 route tests
1d13cd0df Pin gzip negotiation precedence, streamed bodies and Starlette's responders	tests only; with the wildcard briefly made to win over an explicit gzip entry, both precedence cases fail	40 tests
a974a5d5d Say plainly which catalog versions have a search file	the service and Postgres tests failed on the old 503 message	168 unit and 26 Postgres tests
97393e3de Type-check the gzip negotiation module with the v2 code	configuration and documents only	mypy (175 files); rules validate, render and guard
0e6bd7f86 Look up the stored catalog version through the shared helper	refactor, no behavior change	the focused set below

At every commit, ruff passes and the focused unit set passes (the search file, metadata route, metadata service, publication, application and gzip modules: 203 to 268 tests). From the second commit on, the Postgres publication and metadata route tests pass at every commit (50 to 62 tests).

On the final commit:

• make test set (without coverage), under the locked .py 6.10.0 from the lock file: 3,737 passed.
• Postgres tests on a schema recreated at head, grouped as the v2 workflows run them:
  • v2 migration lifecycle and revision tests: 48 passed;
  • after privilege apply and verify: publication and routes 62 passed; user identity and saved-policy view 20 passed; comparison runs and privilege reconciliation 9 passed.
• alembic current --check-heads: c614b6bcccfc (head); alembic check: no new operations.
• ruff, format, mypy (175 files), quality guards and rules validate and guard: clean.
• Rules render and contract export: no change.
• No commit adds a .coverage file or names a private repository.

Review

• Round 1 found two minor bugs.

  • A client that refused gzip (gzip;q=0) still received gzip under the uncompressed form's ETag, because Starlette's middleware compressed the body. Fixed in 5a1bb4d0a, which changes negotiation on every route (above).
  • The file left out the labelled gov node ("Government"), which v3's concept clusters read. Fixed in 7eb8f5080.

  Its quality findings led to the move onto 1.16's caching route class (in the route commit, during the rebase onto 1.16 and 1.17), 6f9ae2ea6, 32566f316, 32939ade8 and 4693c89ed. 76767f76e then applied 1.17's rule for versions stored before catalog format 2.

• Round 2 found no substantiated bugs. Its final-pass findings are fixed in:

  • a5dd07f00: the 304 reads only the hash;
  • c8dd1de00: editable_changes is optional and never null, and the shape test compares types;
  • 1d13cd0df: tests for precedence, streamed bodies, Starlette's responders and an editability change in the middle of a value list;
  • a974a5d5d: the 503 message, the changelog fragments and the shared ETag description;
  • 97393e3de: the mypy file list;
  • 0e6bd7f86: 1.17's shared stored-version helper.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 0e6bd7f864.

Verdict: correct and well tested. Two small points on the new gzip negotiation, and one operational consequence to write down. Stacked on #4046, so I reviewed only v2plan/1f-117-parameter-children...v2plan/1f-118-search-file. No must items.

1. should policyengine_api/data/v2/catalog/publication_parameter_search_file.py:109 together with the 503 for "no file of the current format": any change to how the file is built (an editable fix, a new field) has only two outcomes. Without a format bump, publication fails on the installed version and the deploy stops. With a bump, every older named version answers 503 until something backfills it, because only the installed version gets a new file. Both are safe, but the second is a visible regression for the app's pinned versions. Worth stating in docs/migration/stage-9-v2-metadata.md, next to the format paragraph, so the next person to touch the builder knows which one they are choosing.
2. nit policyengine_api/content_encoding.py:45: the first explicit gzip entry decides, so gzip;q=0, x-gzip returns False even though x-gzip is accepted (checked locally). This is rare in practice; taking the highest quality across both names would match RFC 9110.
3. nit policyengine_api/fastapi_routes/v2/metadata/search_file_response.py:74: a client that does not accept gzip makes the route decompress the full 19.9 MB US file in memory on every 200. Browsers always accept gzip, so this only affects scripts, but with 6 concurrent requests an instance holds about 120 MB for it. A cap or a streamed decompression would bound it.

Checked and fine: the gzip response sets Content-Encoding, so the middleware passes it through untouched; the ETags differ per representation and Vary includes Accept-Encoding; If-None-Match reads only the hash; hashing the uncompressed JSON means a zlib change cannot fail publication. The application-wide middleware change is a correct tightening for v1 too: gzip;q=0 now gets identity.

What I ran: fresh clone, uv sync --frozen --extra dev, then pytest on the changed unit and contract test files: 355 passed, plus a quick check of accepts_gzip on six header values. CI on this head was cancelled (CI complete reports the cancelled jobs, not test failures). I did not run the integration or production-scale publication tests, or v3's search evaluation.

🤖 Generated with Claude Code