Skip to content

UK breakdowns: start in the background, poll through brief failures, time limit, measured estimate, nation withheld counts, map credit - #1427

Merged
PavelMakarchuk merged 3 commits into
mainfrom
fix/uk-local-area-breakdown-followups
Oct 27, 2026
Merged

PavelMakarchuk merged 3 commits into
mainfrom
fix/uk-local-area-breakdown-followups

Conversation

@PavelMakarchuk

@PavelMakarchuk PavelMakarchuk commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1411. Follow-ups to #1398's on-request UK local-area breakdowns.

Changes

  • Polling outlasts brief failures (useUKLocalAreaBreakdown). A computing run keeps polling through up to MAX_FAILED_POLLS (3) failed polls in a row, for example a gateway 502, because the job may still run on the server. The next failure after that, or a failure before the run is known to be computing, still ends it as an error.

  • Time limit. A run that computes for longer than UK_LOCAL_AREA_BREAKDOWN_TIME_LIMIT_MS (30 minutes) ends with an error and Retry. Retry checks the same run again with a fresh clock.

  • Measured duration. UK_LOCAL_AREA_BREAKDOWN_DURATION_MS is now 270 s, from runs on the Public API on 26 October (.py 6.10.0), submission to completion:

    Run Time
    UK-wide, cold worker 267 s (queue 40 s, loading data 90 s, baseline 50 s, reform 60 s, outputs 25 s)
    UK-wide, warm worker 154 s (two runs)
    Wales, warm worker 46 s
    Any repeated request (cache) about 1.5 s

    The API's stages keep a faster run's progress bar honest.

  • Exact withheld counts for nation reports (countWithheldUKAreas). A nation report counts its published areas by ONS code against the nation's areas in the shipped boundary files (UK_AREA_COUNTS_BY_NATION: constituencies E 543, W 32, S 57, N 18; local authorities E 296, W 22, S 32, N 11). A new test in ukBoundaryFiles.test.ts checks these numbers against the files. A count is given only when every published area has a code in the nation; otherwise the note keeps its wording without a number, as before. UK-wide reports keep the existing count against the region metadata.

  • Licence credit in map downloads (UKAreaChoroplethMap). The export ref now wraps the map frame and the ONS / Open Government Licence credit under it, so PNG and SVG downloads include the credit. Tooltip positioning still measures the frame.

Not in this PR (left open on #1411)

  • Keeping a completed breakdown with the report. A saved level is treated as not disclosure-controlled, so it would lose its "not published" note. This needs a design decision first.
  • The two v2 items (old-release wording, keyboard access to the v2 map). v2 report pages stay unchanged until the v3 switch.

Testing

  • New tests: failed polls outlasted then completed; failures beyond the limit end as an error; the time limit ends a run and Retry checks it again; nation counts (Wales count, no areas published, an area of another nation, an area without a code, Scotland in withheldUKAreasOf); nation counts against the boundary files; the export node holds the map and its credit.
  • bun run vitest: 457 files, 5,909 tests pass. bun run check (tsc, ESLint, Prettier) passes.
  • Browser (local, UK flag on): report 4729's Local areas tab computes the breakdown, draws the maps and reads "71 of 650 constituencies are not published"; the credit is inside the export node.

Related: PolicyEngine/policyengine-sim-api#914 (reuse the baseline across breakdown runs).

🤖 Generated with Claude Code

Background start (v3)

UKLocalAreaAutoStart (commit 89a7fadfd) starts the breakdown as soon as a UK-wide or nation report's national result is in, so the user no longer clicks Compute. It renders nothing; the Local areas tab follows the same run through the query key and shows its progress, then the maps, or Retry after an error. Results that already hold both levels, and results from before .py 6.10.0, start nothing. A failed run is not restarted automatically. The overview card now reads "They calculate in the background, in a few minutes." v2 is unchanged.

Checked locally on report 4783: the breakdown=local_areas request went when the overview loaded, and after the run finished, the Local areas tab showed the maps with no click.

Cost: every opened UK-wide or nation report from .py 6.10.0 on now sends one breakdown run (cached by the API after the first). sim-api#914 (baseline reuse) would cut each run by about 50 s.

PavelMakarchuk and others added 2 commits October 26, 2026 10:37
…nt nation withheld areas and credit map downloads

- useUKLocalAreaBreakdown: a computing run outlasts up to three failed polls
  in a row, and fails with Retry once it computes past
  UK_LOCAL_AREA_BREAKDOWN_TIME_LIMIT_MS.
- countWithheldUKAreas: a nation report counts its published areas by ONS
  code against the nation's areas in the boundary files
  (UK_AREA_COUNTS_BY_NATION, checked against the files by a test).
- UKAreaChoroplethMap: the export ref wraps the frame and the boundaries'
  Open Government Licence credit, so PNG and SVG downloads include it.

Part of #1411.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
UK-wide 267 s on a cold worker and 154 s on a warm one, Wales 46 s, on the
Public API on 26 October 2026.

Part of #1411.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policyengine-calculator-next Ready Ready Preview Oct 26, 2026 8:21pm UTC
policyengine-website Ready Ready Preview Oct 26, 2026 8:21pm UTC

Request Review

…akdown-followups

# Conflicts:
#	app/src/tests/unit/data/ukBoundaryFiles.test.ts
@PavelMakarchuk
PavelMakarchuk merged commit e70fd97 into main Oct 27, 2026
10 checks passed
@PavelMakarchuk PavelMakarchuk changed the title UK breakdowns: poll through brief failures, time limit, measured estimate, nation withheld counts, credit in map downloads UK breakdowns: start in the background, poll through brief failures, time limit, measured estimate, nation withheld counts, map credit Oct 27, 2026

This branch was successfully deployed

2 active deployments
Preview – policyengine-calculator-next — 3e7a2faa Deployed Oct 26, 2026 by vercel[bot]
Preview – policyengine-website — 3e7a2faa Deployed Oct 26, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants