Skip to content

App country dimension: surface the UK end-to-end - #50

Merged
MaxGhenis merged 7 commits into
mainfrom
uk/app-country-dimension
Aug 20, 2026
Merged

App country dimension: surface the UK end-to-end#50
MaxGhenis merged 7 commits into
mainfrom
uk/app-country-dimension

Conversation

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

Closes #42.

What was done

Data contract (data/lanes.json + committed copy app/public/data/lanes.json, kept identical)

  • Every lane now carries an explicit "country" field — "US" for existing lanes, "UK" for dwp-takeup, hbai-poverty, hmrc-personal-tax, obr-welfare, ukmod-stats, ukmod-cases — documented in the _schema block. The diff adds only country lines (plus the writer-canonical trailing newline); stage/note/updated untouched. sync_lane_feed merges by key, so pipeline rewrites preserve the field.

App types (app/src/types.ts)

  • New Country type ("US" | "UK"), COUNTRY_LABELS, and a single countryOf() helper: a missing country key always means US, so US-era comparison/populations exports never drop a row. Lane, Row, and PopulationRow gain the optional country field.

Scoreboard views (App.tsx)

  • US/UK toggle in the header scopes the coverage spine, headline, scorecard, divergences and gaps views; switching resets row filters (geography codes don't carry across countries).
  • Empty states: with UK lanes mid-pipeline, the UK view renders a status panel listing each UK lane and its stage from lanes.json — the country is never hidden. The same panel serves any future country with zero rows.
  • Mission control lists every UK lane with its current stage (a "UK pipeline" block that will show stage transitions as lanes move), and running lanes carry a country chip when non-US.
  • ComparisonTable/DivergenceBoard/MissionControl generalize the hardcoded "US" national-geography checks to the row's country code, so UK national rows land in the right buckets when they arrive.
  • Status taxonomy untouched: bucketOf, SPINE_META, and the status chips are country-blind, so pe_gap and concept_mismatch UK cells will render exactly like US ones and stay on the page.

Reform-validation tab

  • States its scope explicitly while single-country: "scope: United States only — … a country filter appears here once claims from another country land". When the populations feed contains a second country, a Country select appears automatically and filters by countryOf.

What remains

  • No UK comparison/populations rows exist yet on main — the UK path currently exercises the empty states and lane listing; the pipeline exports (build_comparison/export_populations) still need to emit country on rows when UK claims land (other branches carry "country": "UK" externals).
  • The scorecard table's external column header/vintage chips are still Urban-specific; they should key off per-country source metadata once a UK instance ships.
  • The page-top provenance stamp describes the Urban instance only (noted in-app).

Verification

  • bun install --frozen-lockfile && bun run lint && bun run build in app/: lint clean (no warnings), tsc -b && vite build pass.
  • PYTHONPATH=. uv run --with pytest pytest tests/ -q from repo root: 127 passed, 4 skipped.
  • data/lanes.json and app/public/data/lanes.json verified byte-identical.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor

Dual-gate review: HOLD on the UK feed-architecture adjudication (#48 vs #32), by design — PopulationRow.country expects a top-level field no current exporter emits (UK claims would silently default US via countryOf()), and the singleton source_meta/pe_bundle provenance stamp can't faithfully represent mixed-country feeds. Verified safe otherwise: missing country defaults US, all 30,004 legacy comparison rows + 270 population rows retained, both lane mirrors preserve all 30 lanes. Re-gate once the ingest architecture lands.

@DTrim99

DTrim99 commented Aug 18, 2026

Copy link
Copy Markdown
Collaborator

Review — App country dimension (UK end-to-end)

No backward-compat/data-loss bug — the load-bearing contract holds: countryOf() (types.ts:22) defaults a missing country to "US", and the row-scoping filter (App.tsx) plus the Divergence/MissionControl national checks were all migrated from === "US" to === countryOf(r), so no legacy US row/lane/population export is dropped. The two committed lanes.json copies match in the diff, and the toggle handles empty-country states via CountryEmptyState.

Should address

  • AttributionPanel.tsx:39,43 still hardcodes geography === "US" / !== "US". This panel is rendered by DivergenceBoard (which now receives UK-scoped data), fetches its own US-only exhibits.json, and ignores the toggle — so under the UK view it renders US attribution unconditionally. It's a separate feed (not a dropped-row bug), but it's the one filter site the PR left unconverted; gate it by country or label it US-only.
  • LANE_FEED append path mints country-less entries (ingest_harvest.py:104, entry = {"id": id, **meta}). sync_lane_feed preserves country on rewrite of existing lanes (good), but any new lane added via DB-append silently defaults to US — so UK lanes must be seeded in lanes.json directly, not via LANE_FEED.
  • No app testscountryOf(), the scoped filter, and empty-state paths are untested. At minimum a countryOf() unit test.

Suggestion

  • The committed app/public/data/lanes.json copy can drift between builds since nothing verifies equality in CI (the predev/prebuild cp regenerates it, but there's no check). A CI equality check or .gitignore on the generated copy would remove the manual-sync hazard (pre-existing, not introduced here).

Reviewed with Claude Code assistance.

@vahid-ahmadi

Copy link
Copy Markdown
Contributor Author

All three should-address items fixed in c5c17e7:

  1. AttributionPanel now takes the selected country (threaded filters.countryDivergenceBoard → panel) and renders nothing outside US — exhibits.json is a US-only feed (SPM poverty, state geographies), so gating beats labeling: the UK view shows no US attribution at all.
  2. Country-less lane appends refused: sync_lane_feed raises if a newly appended entry's meta lacks country; every LANE_FEED entry and the populace-reform-validation caller now carry "country": "US" explicitly. Pinned by two tests (all-entries-carry-country + the append-path raise).
  3. App tests: bun test src runs a countryOf() suite (missing-key defaults to US, explicit pass-through, and the scoping filter never dropping US-era rows). Test files are excluded from the tsc -b build graph so the CI build stays clean.

Suite: python 129 passed / 4 skipped (was 127); bun run lint + bun run build + bun test src all pass.

The lanes.json drift suggestion is left as-is for now (pre-existing, as you noted — predev/prebuild regenerate the copy); happy to add a CI equality check in a follow-up.

The #57-adjudication HOLD stands — these fixes just clear the re-gate queue for whenever the ingest architecture lands.

@DTrim99

DTrim99 commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Re-review — all addressed ✅

  • AttributionPanel now takes { country } and returns null unless country === "US" (US-only exhibits.json feed) — no more unconditional US attribution under the UK toggle.
  • LANE_FEED append path now raises when an entry lacks country, and every entry is tagged.
  • Tests added (types.test.ts: countryOf() default/passthrough + the scoping filter) with a bun test script.

Only cosmetic TS-typing nits left (a string-vs-Country index in ReformValidationView); the lanes.json dual-copy CI-equality check remains a pre-existing nice-to-have. Good to go from my side.

@DTrim99 DTrim99 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed — all findings addressed (AttributionPanel gated by country, LANE_FEED raises on missing country, types.test.ts added). Approving; the remaining TS-typing/lanes.json-dup items are cosmetic/pre-existing.

@MaxGhenis

Copy link
Copy Markdown
Contributor

Heads-up: the UK foundation landed on main today, so this branch's data surfaces have moved under it.

What changed that touches this PR:

  • UK externals → DB ingest path (16,924 claims, five sources) #48 merged — the five UK sources are ingested (15,851 claims + 1,073 Ledger admin-outturn facts), and data/lanes.json now carries five UK lanes at ingested (dwp-takeup, hbai-poverty, hmrc-personal-tax, obr-welfare, ukmod-stats).
  • UC deductions: FRR family re-harvested from primary sources + DB ingest #52 merged — the FRR deductions family (7 claims, sources hm_treasury + dwp) and a sixth UK lane, uk-deductions-frr. Also a change worth knowing about when hacking on the app locally: opening the DB no longer rewrites it (the comparisons view used to be dropped/recreated on every open, which dirtied the committed blob for any reader — ScorecardDB now recreates it only on definition drift).
  • Campaign-UK staging producer: archive -> resolved claim-id joins + 14 reckoner attaches #72 (in final gate) — 14 UK reckoner claims get PE results (run_id campaign-20260802-reckoner-t2), and data/populations.json goes 270 -> 284 rows, so the Reform validation tab will have UK rows with a constructed effective status from the cross-baseline guard (PE current-law result vs the claims' HMRC indexed baseline).

Suggested sequence: once #72 merges (imminent), rebase onto main — export_populations.py, ingest_harvest.py, and both lanes.json copies will conflict, and main's versions carry gate-hardened behavior (the exporter's per-result annotations/baseline labels, sync_lane_feed's lanes parameter contract), so resolving toward main and re-applying the country dimension on top is the safer direction than keeping this branch's versions. After the rebase we'll run the standard dual gate (fable + sol) on the head.

🤖 Generated with Claude Code

vahid-ahmadi and others added 3 commits August 19, 2026 20:36
Adds an explicit country field to data/lanes.json (US for existing lanes,
UK for the six DWP/HBAI/HMRC/OBR/UKMOD lanes) and threads a Country type
through the app's data contracts: lanes, comparison rows and population
rows, with a missing country always read as US so historical exports
never drop. The header gains a US/UK toggle that scopes the coverage
spine, scorecard, divergences and gaps views; UK renders as a pipeline
status panel listing each lane's stage while no UK rows exist, mission
control shows the UK lanes with their stage transitions, and the
reform-validation tab states its US-only scope explicitly until claims
from a second country land (then it grows a country filter). Status
taxonomy is untouched — pe_gap and concept_mismatch render identically
for any country.

Closes #42

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…pended lane entries, countryOf tests

- AttributionPanel takes the selected country and renders nothing outside
  US: exhibits.json is a US-only feed (SPM poverty, state geographies),
  so the UK view no longer shows US attribution unconditionally. Threaded
  via DivergenceBoard from filters.country.
- sync_lane_feed refuses to append a feed entry whose meta lacks
  'country' (countryOf() would silently render it as US); every
  LANE_FEED entry and the populace-reform-validation caller now carry
  country explicitly. Both pinned by tests.
- App unit tests: countryOf() default/explicit/scoping contract under
  `bun test src` (test files excluded from the tsc -b build graph).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebase completion (the branch forked before the whole UK arc — #43-#48,
#52, #71, #72 all landed under it):

- uk-deductions-frr (merged today, unknown to this branch) was the one
  committed lane without a country tag -> UK.
- The UK ingests' appended lane metas now carry country explicitly
  (ingest_uk_externals' five lanes + ingest_uk_deductions' one), so a
  fresh-feed append passes the new sync_lane_feed guard and never files
  a UK lane under the app's missing-key US default.
- app/public/data copies refreshed from data/ (they had drifted to a
  270-row populations.json vs 284) and pinned: new test asserts the
  committed copies byte-match data/, and a second asserts every
  committed lane carries US|UK — the two drift classes this rebase
  surfaced.

Suite: 241 python + 3 bun; oxlint + vite build clean; committed DB
byte-stable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis force-pushed the uk/app-country-dimension branch from c5c17e7 to cedd1b5 Compare August 20, 2026 00:39
@MaxGhenis

Copy link
Copy Markdown
Contributor

Rebased onto main in place (your two commits intact, one completion commit on top) — the foundation had moved four merges under this branch since it forked, so I did the mechanical part rather than leaving it to go stale:

  • The 3-way merge carried your country tags cleanly onto main's 31-lane feed; the one gap was uk-deductions-frr (merged today, so your branch couldn't have known it) — tagged UK.
  • Your sync_lane_feed append guard is exactly right, and it exposed that the UK ingests' own appended lane metas (the five UK externals → DB ingest path (16,924 claims, five sources) #48 lanes + the FRR lane) didn't carry country — they do now, so a fresh-feed append can't file a UK lane under the app's missing-key US default.
  • Took your "nice-to-have" too: app/public/data copies refreshed (they had drifted to a 270-row populations.json vs data/'s 284) plus two pinning tests — committed copies must byte-match data/, and every committed lane must carry US|UK.

Suite: 241 python + your 3 bun tests; oxlint + vite build clean; CI green. Dual gate (sol pass) running next — merges on MERGE-SAFE per the standing sequence. Shout if anything in the completion commit doesn't match your intent.

🤖 Generated with Claude Code

MaxGhenis and others added 4 commits August 19, 2026 21:04
Sol's three blockers on the rebased head, each fixed and verified in
the running app:

1. The populations exporter never emitted country, so all 284 rows —
   including the 14 UK reckoner claims cedd1b5 shipped into the app
   copy — classified as US under countryOf()'s missing-key default.
   Rows now carry country from the claim's own conditions (US: 270,
   UK: 14, pinned in the export integration test); regenerated feed +
   app copy.

2. fmtV prefixed every non-share value with $: currency now follows
   value_kind (gbp* -> £, usd* -> $, counts unsymboled; percent and
   index kinds render properly). uk_hmrc and the other UK source slugs
   get display labels.

3. The header toggle now owns the country dimension everywhere:
   MissionControl takes country — lanes/backlog scope to it, and the
   US-feed-derived held-out record and freshest divergences render
   only under US, with truthful UK cards in their place (the first UK
   results are on the validation tab, held-out relationship on a
   constructed basis; a UK record lands when UK comparisons join the
   comparison feed). ReformValidationView takes country and scopes
   rows, source/status counts, and its summary prose to it — its
   internal country select is gone (one owner).

Coverage: CI now runs the bun tests (they existed but never ran).

Verified in the built app: US view unchanged; UK view shows the 7-lane
pipeline, honest cards, and 14 HMRC rows at £ values with constructed
statuses. Suite: 241 python + 3 bun; oxlint + vite build clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sol's three round-2 findings:

1. HIGH — source/status filter state carried across country toggles,
   so a US filter (e.g. Source=JCT) emptied the UK table and vice
   versa. ReformValidationView is now keyed by country in App.tsx:
   the toggle remounts it, so no internal state can survive a country
   switch — the failure class is gone by construction, both
   directions. Verified in the running app on sol's exact repro
   (US -> validation -> JCT -> UK => selects reset, 14 rows render).

2. MEDIUM — MissionControl was US-gated, not country-scoped: its
   record/divergences read ALL comparison rows under US and were
   forced empty under UK, so future UK comparison rows would have
   inflated the US cards while staying hidden under UK. Both now
   compute from countryOf-scoped rows; the explainer cards render on
   emptiness (with the UK-specific prose conditional on UK), numbers
   render whenever a country's rows exist.

3. MEDIUM — the freshest-divergences UK copy repeated the corrected
   false claim ("first UK held-out results"); it now says "when UK
   rows join the comparison feed".

Also per the round's scope note: the provenance stamp is labeled
"external (US comparison)" and the Method tab opens with an explicit
US-scope line (the country toggle does not change either).

Suite unchanged (241 python + 3 bun); oxlint + vite build clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…alification

Sol's round-3 blocker: the cards said "No {country} rows in the
comparison feed" whenever heldOut/freshest were empty — but a feed
slice of consumed-target or valueless rows would make that copy lie.
Both cards are now three-state: numbers when qualifiers exist;
feed-absence copy only when countryRows is actually empty; and honest
"rows exist but none qualify" / "no material national divergences"
states between.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@MaxGhenis
MaxGhenis merged commit 80ad215 into main Aug 20, 2026
2 checks passed
@MaxGhenis
MaxGhenis deleted the uk/app-country-dimension branch August 20, 2026 02:52
MaxGhenis added a commit that referenced this pull request Aug 20, 2026
Sol's three findings:
1. CI now runs `git diff --exit-code` after the determinism builds AND
   after pytest — the builds rewrite tracked feeds (lanes, ledger,
   uk_resolved) as ingest side effects, so the tree must equal HEAD or
   a committed artifact has gone stale relative to the code that
   regenerates it. (App copies are tracked, so the same gate covers
   data/ vs app/public/data drift.)
2. Fresh-clone workflow: tests/conftest.py session fixture builds the
   DB when absent (plain pytest works from a clean clone; no-op in
   CI); scorecard_db/README documents build_db as the canonical flow,
   the from-scratch refusal, and the empty-DB-from-ScorecardDB() trap.
3. pipeline/build_comparison.py now reads the vendored interchange and
   fails loudly when it is missing (it used to silently build empty).

Plus sol's low-risk hardening: build_db refuses any duplicate
(claim_id, computed_at) in pe_results — the exporters use the
autoincrement id as an ORDER BY tie-breaker, so a tie would let
physical insert order leak into populations.json (zero ties today).

Rebased onto post-#50 main (the app-copy refresh commit was superseded
by the merge and dropped). Hash unchanged: 0c63dff1.... Suite: 247;
tree verified clean through build + suite locally.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Surface the UK in the scorecard app: country dimension end-to-end

3 participants