Serve the v2 parameter tree one level at a time, with nested bracket schedules and the reform view

.github/scripts/test_alembic_v2_lifecycle.sh

@@ -4,4 +4,5 @@ set -euo pipefail
 
 uv run --frozen --extra dev pytest -q \
   tests/unit/v2/test_alembic_v2.py \
-  tests/integration/test_alembic_v2_lifecycle.py
+  tests/integration/test_alembic_v2_lifecycle.py \
+  tests/integration/test_alembic_v2_catalog_tables.py

changelog.d/v2plan-1f-117-parameter-children.added.md

@@ -0,0 +1 @@
+Catalog publication writes parameter trees into a new table, parameter_tree_children, built from the installed packages' catalog records for each version: one row per node and parameter under its parent's path, with its label, its position among its siblings, its direct child counts, and whether a reform can change it. A bracket schedule is one node whose children are its brackets, labelled "Bracket 1", "Bracket 2" and so on, and labels take the form the app's v3 workspace shows, with a path without a stored label getting one made from the last part of its name. Publication writes the tree of each version it publishes, and of the installed version when an earlier release stored it with catalog format 2 but without one; its evidence lists them. Other stored versions, and an installed version stored before catalog format 2, have no tree.

changelog.d/v2plan-1f-117-parameter-children.changed.md

@@ -0,0 +1 @@
+/v2/parameters/children reads one level of the stored parameter tree with one indexed read. A bracket schedule is now listed as one node whose children are its brackets, instead of its brackets appearing beside its parent's groups under their raw names, and every child carries its label in the form the app's v3 workspace shows: the stored label with underscores as spaces and its first letter upper-cased, or one made from the last part of its name. A label PolicyEngine.py made for a bracket field or breakdown parameter without one of its own, such as "Standard deduction (Single)", is shown by the last part of its name, as v3 shows it today. reform_changeable=true lists only the children a reform can change, sorted by label, with child counts that count only those, following the rules of the tree the app's v3 workspace builds. The route now reads its query through the shared strict models, so unknown or repeated parameters, an unsupported country and a malformed policyengine_version return 422 (an unsupported country and a malformed version returned 400 before), and a catalog version without a tree returns 503 with a message naming the version.

docs/engineering/skills/alembic-migrations.md

@@ -233,9 +233,10 @@ check the declarations:
 - `api.v2.runtime-privileges-declared`: a unit test reads each connector
   module and fails when it needs a privilege its role isn't declared to hold.
 - `api.v2.runtime-privileges-tested-as-runtime-role`: the cross-database
-  integration tests run the v2 user services as `policyengine_v2_runtime` and
-  the comparison-run services as `policyengine_stage12_test`, holding only
-  their declared privileges.
+  integration tests run the v2 user services and the parameter tree read as
+  `policyengine_v2_runtime`, catalog publication as
+  `policyengine_v2_catalog_writer`, and the comparison-run services as
+  `policyengine_stage12_test`, holding only their declared privileges.
 - `api.v2.deployed-privileges-match`: every deploy runs
   `scripts/reconcile_v2_privileges.py apply` right after the schema upgrade,
   then `verify`, before the catalog is seeded.
@@ -267,6 +268,8 @@ When a pull request adds or changes a v2 table, or a connector's use of one:
    uv run python scripts/reconcile_v2_privileges.py verify
    uv run pytest -q tests/integration/test_v2_user_identity_persistence.py \
      tests/integration/test_v2_stage12_comparison_persistence.py \
+     tests/integration/test_v2_metadata_routes.py \
+     tests/integration/test_v2_catalog_publication.py \
      tests/integration/test_v2_privilege_reconciliation.py
    ```
 

docs/engineering/skills/testing.md

@@ -107,7 +107,7 @@ The generated-only v2 Alembic chain and disposable Postgres lifecycle:
 ```bash
 uv run pytest tests/unit/v2/test_alembic_v2.py tests/unit/test_alembic_workflows.py -q
 V2_ALEMBIC_DISPOSABLE_TEST=1 V2_MIGRATION_DATABASE_URL="postgresql+psycopg://.../policyengine_v2_alembic_test" \
-  uv run pytest tests/integration/test_alembic_v2_lifecycle.py -q
+  uv run pytest tests/integration/test_alembic_v2_lifecycle.py tests/integration/test_alembic_v2_catalog_tables.py -q
 ```
 
 The disposable lifecycle must start from empty Postgres, upgrade to `head`,
@@ -305,6 +305,7 @@ V2_MIGRATION_DATABASE_URL="postgresql+psycopg://.../policyengine_v2_alembic_test
 uv run pytest \
   tests/integration/test_alembic_mysql_lifecycle.py \
   tests/integration/test_alembic_v2_lifecycle.py \
+  tests/integration/test_alembic_v2_catalog_tables.py \
   tests/integration/test_v2_policy_persistence.py \
   tests/integration/test_v1_policy_dual_write.py \
   tests/integration/test_mysql_policy_dual_write.py \
@@ -379,6 +380,7 @@ V2_MIGRATION_DATABASE_URL="postgresql+psycopg://.../policyengine_v2_alembic_test
 uv run --frozen pytest \
   tests/integration/test_alembic_mysql_lifecycle.py \
   tests/integration/test_alembic_v2_lifecycle.py \
+  tests/integration/test_alembic_v2_catalog_tables.py \
   tests/integration/test_v2_household_persistence.py \
   tests/integration/test_v1_household_dual_write.py \
   tests/integration/test_mysql_household_spm_dual_write.py -q
@@ -436,6 +438,7 @@ V2_ALEMBIC_DISPOSABLE_TEST=1 \
 V2_MIGRATION_DATABASE_URL="postgresql+psycopg://.../policyengine_v2_alembic_test" \
 uv run --frozen pytest \
   tests/integration/test_alembic_v2_lifecycle.py \
+  tests/integration/test_alembic_v2_catalog_tables.py \
   tests/integration/test_v2_user_identity_persistence.py -q
 ```
 

docs/generated/rules.md

@@ -248,7 +248,7 @@ flowchart LR
 | `api.migration.cross-database-integration` | Cross-database integration tests pass against real MySQL and Postgres. | mechanical | enforced | yes | CI job `v2-integration-check` | `pr.yml:v2-integration-check` | — |
 | `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
 | `api.v2.runtime-privileges-declared` | Every v2 table has an entry for each managed database role in policyengine_api/data/v2/privileges.py, and every privilege a v2 connector module needs (from its reads, creates, updates or deletes module name and its row locks) is declared for the role that runs it. | mechanical | enforced | yes | pytest `tests/unit/v2/test_privilege_manifest.py` | `pr.yml:test` | yes |
-| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services as policyengine_v2_runtime and the comparison-run services as policyengine_stage12_<environment>, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
+| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, and the comparison-run services as policyengine_stage12_<environment>, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |
 
 ```mermaid
 flowchart LR

docs/migration/stage-9-v2-metadata.md

@@ -271,6 +271,64 @@ Review the fallback summary on every new PolicyEngine.py release. A changed
 count may identify an upstream change to the release's datasets even when
 publication otherwise succeeds.
 
+### Parameter tree
+
+Publication also writes parameter trees into `parameter_tree_children`, which
+`/v2/parameters/children` reads one level at a time. A tree is built from a
+catalog version's parameter and node records: every node and parameter is one
+row under its parent's path, a `[` starts a child as a `.` does (so a bracket schedule is one node
+whose children are its brackets, labelled "Bracket 1", "Bracket 2" and so on),
+and labels take the form the app's newer interface shows in every view: the
+stored label with underscores as spaces and its first letter upper-cased, or,
+without one, the last part of the name in that form (so an unlabelled schedule
+`...rates.uk` is "Uk"). A label PolicyEngine.py made for a parameter without
+one also gets the last part of the name, as v3 shows it today: a bracket field
+labelled from its schedule ("Income Tax main rates (bracket 1 rate)" is
+"Rate") and a breakdown parameter labelled from its breakdown parent
+("Standard deduction (Single)" is "SINGLE"). A breakdown label is recognised
+when it is an ancestor's stored label followed by one value per level below
+it, each the key, a label the catalog's enum variables give the key, or a
+dimension label followed by the key. A label the package writes in one of
+those forms is shortened too, such as the UK's "Maximum LHA (category A)",
+shown as "A". The catalog's other reads keep the label. Each row also records its position among all its
+siblings and, if a reform can change it, among the siblings a reform can
+change: a parameter under `gov` whose economy or household flag is set and whose path names
+neither `taxsim`, `gov.abolitions` nor `pycache`, and every node above such a
+parameter. These are the rules of the tree the app's newer interface builds
+from v1 metadata. Trees are written only for versions with catalog format 2,
+so every parameter in a tree has both flags.
+
+In the same transaction, the command writes the tree of each version in the
+catalog it publishes, which comes from the installed packages, when that
+version has none: the versions it publishes now, and the installed version when
+an earlier release stored it with catalog format 2 but without a tree. An
+installed version stored before catalog format 2 gets none, since its parameter
+rows store no flags. The rows are built from the
+catalog's own parameter and node records; only the stored parameters' IDs are
+read, by name, so the rows reference them, after publication has checked that
+the stored rows match the catalog. This is a row-only, idempotent insert of
+package-derived rows, which the deployment-seeder passage of the Alembic skill
+allows. The evidence's `parameter_trees_written` lists each version written,
+with its model, version and row count, and publication fails if a version in
+the catalog with catalog format 2 is still without a tree when it finishes.
+
+Other stored versions get no tree. Writing one for them would derive rows from
+what earlier releases stored, a data migration that the Alembic skill's rules
+leave to a human migration decision, so it is not done; such a version gets a
+tree only if a person decides on that migration.
+
+`/v2/parameters/children` lists one level with one indexed read:
+`parent_path` names the node ("" for the top level), and
+`reform_changeable=true` keeps only the children a reform can change, sorted
+by label, with child counts that count only those. Without it, children are
+in path order, with a schedule's brackets in bracket order. A node carries
+its child count, a parameter its summary. An unknown parent path has no
+children. The route reads its query through the shared strict query models,
+so unknown or repeated parameters, an unsupported country and a malformed
+`policyengine_version` return 422 (the last two returned 400 before). A
+catalog version without a tree returns 503 with a message that names the
+version.
+
 ## Validation evidence and retry
 
 Successful JSON evidence contains:

migrations/v2/versions/3869eb814a90_store_each_catalog_version_s_parameter_.py

@@ -0,0 +1,115 @@
+"""store each catalog version's parameter tree
+
+Revision ID: 3869eb814a90
+Revises: e86f19a6c4da
+Create Date: 2026-11-04 01:03:34.038237
+Generation: uv run alembic -c alembic-v2.ini revision --autogenerate
+"""
+
+from typing import Sequence, Union
+
+from alembic import op
+import sqlalchemy as sa
+import sqlmodel
+
+
+revision: str = "3869eb814a90"
+down_revision: Union[str, None] = "e86f19a6c4da"
+branch_labels: Union[str, Sequence[str], None] = None
+depends_on: Union[str, Sequence[str], None] = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.create_table(
+        "parameter_tree_children",
+        sa.Column("id", sa.Uuid(), nullable=False),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.text("now()"),
+            nullable=False,
+        ),
+        sa.Column("tax_benefit_model_version_id", sa.Uuid(), nullable=False),
+        sa.Column(
+            "parent_path", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=False
+        ),
+        sa.Column("path", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=False),
+        sa.Column("position", sa.Integer(), nullable=False),
+        sa.Column(
+            "label", sqlmodel.sql.sqltypes.AutoString(length=512), nullable=False
+        ),
+        sa.Column(
+            "child_type", sqlmodel.sql.sqltypes.AutoString(length=16), nullable=False
+        ),
+        sa.Column("child_count", sa.Integer(), nullable=True),
+        sa.Column("reform_position", sa.Integer(), nullable=True),
+        sa.Column("reform_child_count", sa.Integer(), nullable=True),
+        sa.Column("parameter_id", sa.Uuid(), nullable=True),
+        sa.Column("economy", sa.Boolean(), nullable=True),
+        sa.Column("household", sa.Boolean(), nullable=True),
+        sa.CheckConstraint(
+            "(child_type = 'parameter') = (parameter_id IS NOT NULL)",
+            name=op.f("ck_parameter_tree_children_parameter_reference"),
+        ),
+        sa.CheckConstraint(
+            "child_type IN ('node', 'parameter')",
+            name=op.f("ck_parameter_tree_children_child_type"),
+        ),
+        sa.ForeignKeyConstraint(
+            ["parameter_id"],
+            ["parameters.id"],
+            name=op.f("fk_parameter_tree_children_parameter_id_parameters"),
+            ondelete="CASCADE",
+        ),
+        sa.ForeignKeyConstraint(
+            ["tax_benefit_model_version_id"],
+            ["tax_benefit_model_versions.id"],
+            name=op.f(
+                "fk_parameter_tree_children_tax_benefit_model_version_id_tax_benefit_model_versions"
+            ),
+            ondelete="CASCADE",
+        ),
+        sa.PrimaryKeyConstraint("id", name=op.f("pk_parameter_tree_children")),
+        sa.UniqueConstraint(
+            "tax_benefit_model_version_id",
+            "path",
+            name="uq_parameter_tree_children_model_version_path",
+        ),
+    )
+    op.create_index(
+        "ix_parameter_tree_children_listing",
+        "parameter_tree_children",
+        ["tax_benefit_model_version_id", "parent_path", "position"],
+        unique=True,
+    )
+    op.create_index(
+        op.f("ix_parameter_tree_children_parameter_id"),
+        "parameter_tree_children",
+        ["parameter_id"],
+        unique=False,
+    )
+    op.create_index(
+        "ix_parameter_tree_children_reform_listing",
+        "parameter_tree_children",
+        ["tax_benefit_model_version_id", "parent_path", "reform_position"],
+        unique=True,
+    )
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_index(
+        "ix_parameter_tree_children_reform_listing",
+        table_name="parameter_tree_children",
+    )
+    op.drop_index(
+        op.f("ix_parameter_tree_children_parameter_id"),
+        table_name="parameter_tree_children",
+    )
+    op.drop_index(
+        "ix_parameter_tree_children_listing", table_name="parameter_tree_children"
+    )
+    op.drop_table("parameter_tree_children")
+    # ### end Alembic commands ###