UK consumer feed tooling: suite, facts-only artifacts, inventory roots (#135 PR B)

[juaristi22]

Summary

PR B, closes #135, stacked on #181 (PR A). Three enablers for the microcosm handoff now that selection contracts live consumer-side (#166):

1. UK suite — chronicle build-bundle --suite uk over a curated UK_BUNDLE_SOURCES tuple (66 packages across the UK source prefixes), with a drift test asserting the tuple equals the prefix-derived set from SOURCE_PACKAGE_ALIASES — a new UK package cannot be silently omitted. Documented decision per the issue: --year is inert for the UK suite (every UK package is year-pinned via artifact_year + literal periods); the US off-year skip behavior is untouched and out of scope.
2. Facts-only consumer artifacts — the zero-profile guard in policyengine_chronicle/consumer.py is relaxed to return {}: a facts-only invocation writes "profiles": {} and an empty profiles/ directory, which load_consumer_artifact round-trips. This is the artifact shape the Adjudicate: do target profiles and measurement bindings belong in Chronicle? The boundary records disagree #166 ruling leaves Chronicle with; the pinned refusal test became a facts-only round-trip test. Retire the target-profile surface: Chronicle is facts-only (#166 ruling) #172 (profile-surface retirement) inherits the rest.
3. Inventory de-hardcode — db/pe_source_inventory.py loses the /Users/maxghenis/... roots (both hardcode sites): roots come from --pe-us-root/--pe-uk-root or LEDGER_PE_US_DATA_ROOT/LEDGER_PE_UK_DATA_ROOT (the source_package.py env convention), are required only for the requested --jurisdiction, and fail loudly naming both mechanisms when unset (distinct message for set-but-nonexistent). The sharp edge is closed: an empty spec list now refuses to prune_source_artifacts instead of wiping prior ingestion.

End-to-end evidence (run against this branch)

uv run chronicle build-bundle --suite uk --out /tmp/chronicle-uk --replace       # 107,515 UK facts
uv run chronicle build-consumer-artifact --facts /tmp/chronicle-uk --out /tmp/chronicle-uk-artifact --replace

Artifact manifest: "profiles": {}, facts_sha256 8504554a69b6d5a4…, manifest_sha256 ddb11bffb0e23e16… — and microcosm's load_ledger_consumer_artifact loads it hash-pinned (fact_row_count 107515, provenance carries both hashes). That is the exact feed microcosm#622 compiles the re-homed contract against.

CI cost note: no full UK bundle build was added to CI — the suite is covered by the membership/drift test plus small-fixture artifact tests; the build above is the manual evidence run.

Chronicle Governance

• Approved Chronicle agent role: ledger-contract-maintainer (consumer artifact semantics); the db/** inventory work is the maintainer tooling UK enablement tooling: source-inventory de-hardcode, UK consumer-artifact suite, legacy ETL cleanup #135's header sanctions.
• Deterministic checks run: schema validation, consumer-contract validation, package import compatibility, raw-facts boundary validation — full suite uv run pytest -q.
• LLM judge verdicts:
  • ledger-source-fidelity: n/a (no source packages touched)
  • ledger-target-profile: n/a
  • ledger-contract: pending review
  • ledger-boundary: pending review

Tests

13 inventory tests (cleared-env failure naming the env var, env resolution, jurisdiction scoping, nonexistent-path message, prune refusal), suite membership/drift + CLI tests, facts-only artifact round-trip. Implemented by Codex under the test-iteration protocol with zero deviations; verified twice (Codex + independent re-run): ruff clean, full suite 634 passed, 1 skipped.

🤖 Generated with Claude Code

[vahid-ahmadi]

All three enablers verified in the diff, and the targeted suites pass for me on the stacked branch.

The facts-only relaxation is the piece I looked at hardest, because loosening a guard is where a ruling's implementation can quietly overshoot: the change is exactly three lines — _load_profiles returns {} instead of raising when neither ids nor paths are given — and the artifact keeps its shape ("profiles": {} plus the empty directory) so load_consumer_artifact round-trips without a special case. Converting the pinned refusal test into a facts-only round-trip test rather than deleting it means the old guarantee's replacement is pinned, not just the old guarantee removed. That's the minimal faithful implementation of what #166 leaves Chronicle being; the rest correctly waits for #172.

The UK suite's drift test — the curated UK_BUNDLE_SOURCES tuple asserted equal to the prefix-derived set from SOURCE_PACKAGE_ALIASES — is the closed-world move this repo keeps getting right: a new UK package can't be silently omitted from the feed, and the tuple stays reviewable. Documenting --year as inert for the UK suite (year-pinned packages) instead of wiring a dead parameter is honest; the one thing I'd add is a loud refusal if someone passes a non-default --year with --suite uk, since an inert flag that accepts input is a smaller version of the trap the empty-selector-list fix just closed on microcosm#707. De-hardcoding the inventory roots out of db/pe_source_inventory.py removes the last personal-machine path from the tree — overdue and welcome.

With #181 this completes #135's PR pair as scoped. Ready from my side on both.

[juaristi22]

Codex adversarial review found two issues, both addressed in the latest push.\n\n- High: the branch diff against current main appeared to remove the shipped US SOI county, Census PEP county, and USDA SNAP FY2025 monthly source packages from the default bundle. This was caused by PR 186 being stacked on PR 181 before Chronicle #185 landed, not by a UK feed design decision. The branch is now rebased onto #185/current base, and the US package files, manifests, aliases, loader, tests, and bundle coverage are preserved.\n- Medium: load-source-files with the default all jurisdiction failed when only one PE root was configured. pe_source_specs now uses the configured jurisdiction roots and skips an unrelated unconfigured root, while still failing clearly when no requested root is configured or an explicit path is invalid. Tests cover UK-only and US-only partial configurations.\n\nVerification after the fix: 15 source-inventory tests passed, 8 lightweight bundle/CLI tests passed, Ruff passed, and compile/diff checks passed. The heavyweight merged-bundle test was attempted; 17 tests passed before it was interrupted after 12m34s while parsing the restored large US package corpus, with no assertion failure observed.

[juaristi22]

Follow-up scope: retire the remaining UK HMRC ETL without losing targets

Before deleting db/etl_hmrc.py, adjudicate every legacy HMRC_DATA series. Each row must either be ported to a Chronicle source package with the publisher's exact concept, unit, universe, and period, or be recorded as an explicit gap in docs/pe-uk-source-checklist.md. No legacy target is deleted merely because Microcosm does not currently select it.

Preliminary official source candidates found:

• Income Tax liabilities, taxpayer counts, marginal-rate counts, and income totals: HMRC Income Tax liabilities statistics and Personal Incomes Statistics 2021 to 2022.
• National Insurance: HMRC National Insurance contributions statistics.
• Capital Gains Tax: validate/reuse the existing hmrc-cgt-statistics-2025 package against HMRC CGT Table 1.
• Inheritance Tax and receipts measures: HMRC tax receipts and NICs statistics, with liability-versus-receipt semantics checked explicitly.
• Child Benefit caseload: HMRC Child Benefit Statistics August 2022; expenditure should come from an official expenditure/receipts table rather than the legacy hardcoded value.
• Universal Credit, State Pension, Housing Benefit, and Pension Credit: DWP Benefit expenditure and caseload tables 2022, supplemented by the relevant DWP caseload/take-up tables where the aggregate table does not expose the required measure.

Implementation gate:

1. Add or extend source packages and provenance tests for every sourceable series.
2. Update docs/pe-uk-source-checklist.md for every legacy row, including exact package/record-set mapping or an explicit gap reason.
3. Verify the UK facts-only bundle and Microcosm consumer contract remain intact.
4. Only then remove db/etl_hmrc.py, its tests, loader exports, and the hmrc target-loading CLI branch.

The existing hmrc-cgt-statistics-2025 package appears to cover the CGT family already; the other rows need package-level validation before deletion.

[juaristi22]

Follow-up design constraint for the HMRC ETL cleanup:

Before adding any replacement package or deleting db/etl_hmrc.py, audit every existing HMRC package and its emitted facts. A matching variable/year/unit is not sufficient to call two facts duplicates. The comparison must include the canonical concept and source concept, unit and scale, geography/entity/universe, period type and period, aggregation, assertion (observation versus source_projection), source vintage/artifact year, and record_set_spec_id/source record identity.

In particular:

• A publisher measurement and that publisher's projection may coexist only when their assertion and source record/vintage make the distinction explicit. Any Microcosm selector that could see both must select the intended assertion or fail with an ambiguity; no silent value choice.
• Preserve the publisher's definition year and period_type. Tax year 2023, fiscal year 2023, calendar year 2023, and a 2023 publication vintage are different dimensions. artifact_year is not a substitute for the fact period.
• For every relevant source package, port all publisher-provided periods in 2023-2026, even if the current Microcosm contract materializes only one of them. The contract's selected target years must not determine which facts exist in Chronicle.
• The existing hmrc-cgt-statistics-2025 package already carries the CGT historical series through tax year 2023, so the implementation must reuse/extend that package after semantic comparison rather than create duplicate CGT facts. The same audit applies to the existing SPI and salary-sacrifice packages.
• Add tests for semantic duplicate detection and exact consumer resolution, including an observation/projection pair and same-source facts across multiple definition years.

Update docs/pe-uk-source-checklist.md with package/record-set coverage and any genuine non-equivalence or source gap. Deleting the HMRC ETL remains gated on this inventory, the 2023-2026 coverage audit, unambiguous facts-only bundle resolution, Microcosm loading, and the final boundary tests.

[juaristi22]

Implemented in commit f5191bd.

• Removed db/etl_hmrc.py, its tests, loader export, UK jurisdiction export, and hmrc CLI source.
• Reused the existing hmrc-cgt-statistics-2025 package for the publisher CGT series across tax years 2021–2023; no legacy hardcoded CGT values were duplicated.
• Added HMRC package metadata tests covering source concept, unit, period type/year, assertion, vintage, and record-set identity.
• Updated the UK checklist with exact package coverage or explicit source-gap/non-equivalence decisions for every legacy HMRC_DATA series.
• Added the 2023–2026 definition-year and observation/projection identity gate to the checklist.
• Restored dwp-uc-deductions-march-2025-february-2026 to the curated UK bundle and refreshed its existing geography baseline.

Verification: Ruff passed; the non-bundle suite passed (674 passed, 1 skipped); focused HMRC, boundary, and UK bundle-membership tests passed. The heavyweight merged-bundle test built the artifact but exceeded the local runtime budget before terminal completion.

[vahid-ahmadi]

Re-reviewed for the HMRC retirement in f5191bd, since that landed after my earlier review and is a materially different kind of change from the three enablers I signed off on. It holds up, and the discipline is stronger than PR A's.

The thing I checked hardest is the one that kills retirement PRs: dangling consumers. Clean — the loader export, UK jurisdiction export, and hmrc CLI source all go with the module, and the only surviving etl_hmrc references in the tree are the legacy-row strings inside test_chronicle_boundaries.py, which is the gap register asserting itself. That's the right inversion: the deleted code's inventory is now test-pinned, so a row can't quietly disappear from the checklist later. 167 tests pass for me across the boundaries, source-package, consumer, and alias-drift suites.

The refusal to backdate is the correct call and worth naming. hmrc-spi-income-bands-2023-24 would have been the tempting cover for the legacy taxpayers/total_income rows — same publisher, same variable names, adjacent years — and "must not be silently backdated or treated as the same band universe" is exactly right: a 2021-22 count and a 2023-24 band count are different facts, and coverage by name-similarity is how a facts layer starts lying. Same for CGT, where reusing hmrc-cgt-statistics-2025 for the publisher series while explicitly not duplicating the legacy hardcoded amounts ("their provenance and values do not match the Table 1 liabilities") keeps the package as the single source and leaves the unexplained numbers dead. Every row landing as "explicit source gap or non-equivalence" rather than a stretched mapping is the honest outcome; a retirement that ported nothing but recorded everything is a better result than one that ported plausibly.

The identity-and-period gate reads as the generalization of the UC basis problem from #727/#701 — "an observation and a publisher projection for the same variable, year, and unit are distinct facts and may coexist only when the consumer selector distinguishes their assertion; no selector may silently choose between them" is precisely the no-silent-value-choice rule that the UC monthly series needed and didn't have. Codifying it in the checklist so it binds future packages rather than living in one PR thread is the right home for it. The artifact_year clarification (publication vintage ≠ fact period) closes the other half of that class.

Two notes carried forward, neither blocking:

• My earlier --year suggestion still stands and is now slightly more pointed with the suite surface larger: --suite uk accepting a --year it ignores is a silent no-op on a flag the US path treats as meaningful. A refusal when a non-default year is passed with --suite uk costs one line and removes the trap.
• The bundle constants moved again in this PR ("hmrc": 20533 and the geography baseline refresh). That's the third re-measurement this wave, which is the case #157 makes — and with the HMRC retirement now in scope too, I'd land that refactor before NZ rather than after.

Ready from my side on the expanded scope.