Serve a compact parameter search file for each catalog version at /v2/parameters/search-file

changelog.d/v2plan-1f-118-search-file.added.md

@@ -0,0 +1 @@
+Catalog publication writes a parameter search file for each catalog version it publishes into a new table, parameter_search_files, built from the installed packages' catalog: every parameter a reform can change with its path, stored label, unit, description and whether a draft can edit its current-law value (with the dates from which that changes), the stored labels of the other parameters under gov, and gov and every labelled node under it, as gzip-compressed JSON with its SHA-256. A new route, /v2/parameters/search-file, serves a version's file with a strong ETag, 304 revalidation and long-lived caching for a named version, as gzip to clients that accept it. Publication writes a file for each catalog format 2 version it publishes or runs with installed, so a version stored before catalog format 2, or stored before search files existed and not installed since, has none and returns 503 with a message naming it.

changelog.d/v2plan-1f-118-search-file.fixed.md

@@ -0,0 +1 @@
+Responses are gzip-compressed according to the qualities in Accept-Encoding instead of a search for the text "gzip": a client that refuses gzip (for example gzip;q=0) now receives an uncompressed response, and `Accept-Encoding: *` or a differently cased value such as GZIP now receives gzip, where both used to receive an uncompressed response.

docs/engineering/migration-contracts.md

@@ -8,7 +8,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | --- | ---: |
 | route group count | 9 |
 | workflow count | 16 |
-| request count | 92 |
+| request count | 93 |
 | db entity count | 6 |
 | sim flow count | 3 |
 
@@ -213,6 +213,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
 | `GET` | `/v2/datasets/{dataset_id}?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
 | `GET` | `/v2/regions/{region_id}?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
 | `GET` | `/v2/regions/by-code/state/ca?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.item` |
+| `GET` | `/v2/parameters/search-file?country_id=us` | 200 | `metadata` | none | `format`, `country_id`, `policyengine_version`, `parameters`, `other_parameters`, `nodes` |
 | `GET` | `/v2/tax-benefit-models/by-country/us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.model`, `result.model_version` |
 | `GET` | `/v2/economy-options?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.current_law_id`, `result.region`, `result.time_period`, `result.datasets` |
 | `GET` | `/v2/model-catalog?country_id=us` | 200 | `metadata` | none | `status`, `message`, `result.policyengine_version`, `result.catalog_format`, `result.entities`, `result.basic_inputs`, `result.regions`, `result.datasets`, `result.economy_options` |

docs/generated/migration_contracts.json

@@ -1,7 +1,7 @@
 {
   "metadata": {
     "db_entity_count": 6,
-    "request_count": 92,
+    "request_count": 93,
     "route_group_count": 9,
     "sim_flow_count": 3,
     "workflow_count": 16
@@ -1586,6 +1586,21 @@
             "result.item"
           ]
         },
+        {
+          "expected_status": 200,
+          "headers": [],
+          "method": "GET",
+          "path": "/v2/parameters/search-file?country_id=us",
+          "route_group": "metadata",
+          "stable_response_fields": [
+            "format",
+            "country_id",
+            "policyengine_version",
+            "parameters",
+            "other_parameters",
+            "nodes"
+          ]
+        },
         {
           "expected_status": 200,
           "headers": [],

docs/generated/rules.md

@@ -294,7 +294,7 @@ flowchart LR
 | `api.quality.command-line-modules-ruff-exemptions` | Application modules that also run as command-line programs (if __name__ == "__main__") may print their output (T201); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of T201 for `policyengine_api/{data/v2/catalog/initialization,data/v2/household_migration_qualification,data/v2/policy_migration_qualification,data/v2/privilege_reconciliation,data/v2/user_association_migration_qualification,worker_spm_release}.py` | `pr.yml:lint` | — |
 | `api.quality.command-line-scripts-ruff-exemptions` | Command-line scripts under scripts/ may print their output, run other programs, fetch fixed URLs, stamp today's date and pick random samples (T201, S603, S607, S310, DTZ011, S311); every other ruff rule applies to them. | mechanical | enforced | yes | ruff per-file ignore of DTZ011, S310, S311, S603, S607, T201 for `scripts/**` | `pr.yml:lint` | — |
 | `api.quality.environment-variables-load` | Required environment variables and credentials load in a deployed-like environment. | mechanical | enforced | yes | CI job `test_env_vars` | `pr.yml:test_env_vars` | — |
-| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, and the user sign-in and account-deletion libraries pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:typecheck-v2` | — |
+| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, the gzip negotiation, and the user sign-in and account-deletion libraries pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:typecheck-v2` | — |
 | `api.quality.ruff-format` | All Python files are formatted with ruff format. | mechanical | enforced | yes | ruff format | `pr.yml:lint` | — |
 | `api.quality.ruff-lint` | All Python code passes ruff check with every ruff rule selected (ALL), with no ratchet, except the ignores in pyproject.toml: COM812 and ISC001, which conflict with ruff format, and D203 and D213, which conflict with the kept D211 and D212. | mechanical | enforced | yes | ruff check (ALL) | `pr.yml:lint` | — |
 | `api.quality.runtime-evaluated-models-resolve` | Every package model is one ruff treats as runtime-evaluated (a base resolves, through the import in its own module, to a name in runtime-evaluated-base-classes, or is a same-module class that does), and every Pydantic model and SQLAlchemy mapper resolves its annotations once the package is imported. | mechanical | enforced | yes | pytest `tests/unit/test_runtime_evaluated_annotations.py` | `pr.yml:test` | **unproven** |

docs/migration/stage-9-v2-metadata.md

@@ -354,6 +354,54 @@ so unknown or repeated parameters, an unsupported country and a malformed
 catalog version without a tree returns 503 with a message that names the
 version.
 
+### Parameter search file
+
+Publication also writes the parameter search file of each catalog version it
+publishes into `parameter_search_files`: one gzip-compressed JSON file per
+version and file format, with the SHA-256 of the uncompressed JSON. The file
+holds what the app's newer interface's parameter search reads:
+
+- every parameter a reform can change (the parameter tree's rule) with its
+  path, stored label, unit, description, and whether a draft can edit its
+  current-law value: a number, a yes/no value or an infinite threshold. Where
+  that changes over time (a bracket that stops applying has a null value from
+  that date), `editable` describes the earliest value and `editable_changes`
+  lists the dates from which it changes, so the file does not depend on the
+  day it was built;
+- the path and stored label of every other parameter under `gov`, which the
+  search's concept clusters, acronym aliases and member labels read;
+- the path and stored label of `gov` and every node under it that has a stored
+  label, for breadcrumbs, schedule names and concept clusters.
+
+Labels are stored labels, or null: the search treats a label that only
+restates its path's last key as a placeholder, so its reader applies v1's
+fallback (the key with underscores as spaces, "bracket N" for a bracket).
+
+The file is built from the installed packages' catalog, never from rows an
+earlier release stored, so publication writes one only for the versions it
+publishes: each new version, and the installed version when an earlier
+release stored it before search files existed. A version that is neither has
+no file until a person decides otherwise. An installed version stored before
+catalog format 2 (`catalog_format` null) gets none either: it has no stored
+economy or household flags, so a file built from the installed catalog's
+flags would contradict its own parameter rows. Writing only inserts; a stored file
+of the same format must equal the one built from the installed packages, and
+publication fails if it differs or if a version it publishes is left without a
+file. A later file format adds a row for the version instead of changing the
+stored one. The evidence's `parameter_search_files_written` lists each file
+written with its model, version and uncompressed and compressed sizes in bytes.
+
+`/v2/parameters/search-file` (query `country_id` and optional
+`policyengine_version`) serves the selected version's file of the current
+format. A client that accepts gzip receives the stored bytes with
+`Content-Encoding: gzip`; any other client receives the uncompressed JSON. The
+body is the file itself, not the `status`/`result` envelope. Each
+representation has its own strong `ETag`, made from the SHA-256 of the
+uncompressed JSON (`"<sha256>"`, or `"<sha256>-gzip"` for the gzip one). The
+cache lifetime, `Vary` and the 304 for a matching `If-None-Match` are those of
+every metadata route. A version without a file of the current format returns
+503.
+
 ## Validation evidence and retry
 
 Successful JSON evidence contains:
@@ -366,6 +414,8 @@ Successful JSON evidence contains:
   kept;
 - the model, version and row count of each catalog version whose parameter
   tree was written;
+- the model, version and uncompressed and compressed sizes of each parameter
+  search file written;
 - elapsed publication time; and
 - a success outcome.
 

migrations/v2/versions/c614b6bcccfc_store_each_catalog_version_s_parameter_.py

@@ -0,0 +1,64 @@
+"""store each catalog version's parameter search file
+
+Revision ID: c614b6bcccfc
+Revises: 3869eb814a90
+Create Date: 2026-10-26 16:58:05.350610
+Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+
+
+revision: str = "c614b6bcccfc"
+down_revision: Union[str, None] = "3869eb814a90"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "parameter_search_files",
+        sa.Column("id", sa.Uuid(), nullable=False),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.text("now()"),
+            nullable=False,
+        ),
+        sa.Column("tax_benefit_model_version_id", sa.Uuid(), nullable=False),
+        sa.Column("file_format", sa.Integer(), nullable=False),
+        sa.Column("compressed_content", sa.LargeBinary(), nullable=False),
+        sa.Column(
+            "content_sha256",
+            sqlmodel.sql.sqltypes.AutoString(length=64),
+            nullable=False,
+        ),
+        sa.Column("uncompressed_size", sa.Integer(), nullable=False),
+        sa.Column("compressed_size", sa.Integer(), nullable=False),
+        sa.ForeignKeyConstraint(
+            ["tax_benefit_model_version_id"],
+            ["tax_benefit_model_versions.id"],
+            name=op.f(
+                "fk_parameter_search_files_tax_benefit_model_version_id_tax_benefit_model_versions"
+            ),
+            ondelete="CASCADE",
+        ),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_parameter_search_files")),
+        sa.UniqueConstraint(
+            "tax_benefit_model_version_id",
+            "file_format",
+            name="uq_parameter_search_files_model_version_format",
+        ),
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_table("parameter_search_files")
+    # ### end Alembic commands ###

policyengine_api/asgi_factory.py

@@ -13,11 +13,11 @@
 from fastapi.exceptions import RequestValidationError
 from starlette.datastructures import MutableHeaders
 from starlette.middleware.cors import CORSMiddleware
-from starlette.middleware.gzip import GZipMiddleware
 from starlette.responses import PlainTextResponse, Response
 from starlette.routing import Match, Mount
 
 from policyengine_api.constants import VERSION
+from policyengine_api.content_encoding import NegotiatedGZipMiddleware
 from policyengine_api.fastapi_routes.dependencies import NativeRouteDependencies
 from policyengine_api.fastapi_routes.health import (
     build_core_health_router,
@@ -457,7 +457,8 @@ def create_asgi_app(
     # Cloud Run because it does not have an nginx sidecar to handle gzip.
     # Measured on the real payload: level 9 = 5.5s -> 9.0MB, level 4 = 0.7s ->
     # 9.9MB — 7.5x faster for ~10% larger output.
-    app.add_middleware(GZipMiddleware, minimum_size=1000, compresslevel=4)
+    # It weighs Accept-Encoding qualities, so a client refusing gzip gets none.
+    app.add_middleware(NegotiatedGZipMiddleware, minimum_size=1000, compresslevel=4)
     app.exception_handler(Exception)(_add_headers_to_unhandled_errors)
     app.exception_handler(RequestValidationError)(_typed_v2_request_validation_error)
     app.exception_handler(V2RequestTooLargeError)(_oversized_v2_request)

policyengine_api/content_encoding.py

@@ -0,0 +1,70 @@
+"""Gzip negotiation that weighs Accept-Encoding qualities.
+
+Starlette's ``GZipMiddleware`` compresses whenever the text "gzip" appears in
+Accept-Encoding, so ``gzip;q=0`` (an explicit refusal) still gets gzip. The
+application and the routes that send stored gzip bytes decide with
+``accepts_gzip`` instead, so a response's encoding is the one the client
+accepts and every route agrees with the middleware.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Final
+
+from starlette.datastructures import Headers
+from starlette.middleware.gzip import GZipMiddleware, GZipResponder, IdentityResponder
+
+if TYPE_CHECKING:
+    from starlette.types import Receive, Scope, Send
+
+_GZIP_CODINGS: Final = frozenset({"gzip", "x-gzip"})
+
+
+def _quality(parameters: str) -> float:
+    for parameter in parameters.split(";"):
+        name, _, value = parameter.strip().partition("=")
+        if name.strip().lower() == "q":
+            try:
+                return float(value)
+            except ValueError:
+                return 0.0
+    return 1.0
+
+
+def accepts_gzip(accept_encoding: str | None) -> bool:
+    """Return whether an Accept-Encoding value accepts gzip.
+
+    An explicit gzip entry decides; otherwise ``*`` does. A quality of 0
+    refuses the coding, and a coding name must match exactly.
+    """
+    wildcard_quality: float | None = None
+    for entry in (accept_encoding or "").split(","):
+        coding, _, parameters = entry.partition(";")
+        name = coding.strip().lower()
+        if name in _GZIP_CODINGS:
+            return _quality(parameters) > 0
+        if name == "*":
+            wildcard_quality = _quality(parameters)
+    return wildcard_quality is not None and wildcard_quality > 0
+
+
+class NegotiatedGZipMiddleware(GZipMiddleware):
+    """``GZipMiddleware`` that compresses only for clients that accept gzip.
+
+    Responses that already name a Content-Encoding pass through unchanged, as
+    in the parent class.
+    """
+
+    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
+        """Choose the gzip or identity responder by ``accepts_gzip``."""
+        if scope["type"] != "http":
+            await self.app(scope, receive, send)
+            return
+        accept_encoding = Headers(scope=scope).get("accept-encoding")
+        if accepts_gzip(accept_encoding):
+            responder = GZipResponder(
+                self.app, self.minimum_size, compresslevel=self.compresslevel
+            )
+            await responder(scope, receive, send)
+            return
+        await IdentityResponder(self.app, self.minimum_size)(scope, receive, send)