[anth-volk]

Fixes #1026

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

• This PR adds a changelog fragment, changelog.d/<name>.<type>.md, whose type is breaking, added, changed, fixed or removed.

Summary

• Modal delivers a preemption to the running function as KeyboardInterrupt and then runs the same input again, so the job succeeds. policyengine-observability 3.0.4 recorded every operation that ended in KeyboardInterrupt with outcome error at ERROR, and the Public API's daily telemetry report opened issues for them ([telemetry] policyengine-simulation-py: KeyboardInterrupt (b659eaa2004ca456) policyengine-api#4151, #4153, #4154, #4227, #4228).
• policyengine-observability 3.0.7 (Record operations and requests that end in KeyboardInterrupt as interrupted policyengine-observability#44) records an operation, request or span that ends in KeyboardInterrupt with outcome interrupted at WARNING, leaves it out of the error count metric, leaves the span status unset with policyengine.outcome=interrupted, and lets the exception propagate unchanged.
• Requires policyengine-observability >=3.0.7,<4 (was >=3.0.4,<4) in libs/policyengine-simulation-observability, projects/policyengine-simulation-entry, projects/policyengine-simulation-executor (both dependency lists) and projects/policyengine-simulation-gateway, so a later relock cannot drop the fix.
• Relocks the six projects whose uv.lock holds the package with uv lock --upgrade-package policyengine-observability (uv 0.13.6, the repository's pinned release). Each lock diff changes only that package's version, file hashes and specifier; the file hashes match PyPI's. policyengine-simulation-contract and policyengine-stage12-persistence get the package through policyengine-simulation-observability and change only their lock.
• Adds a "Preemptions" section to docs/engineering/skills/observability.md and updates the floor in docs/engineering/skills/type-checking.md.

What still writes ERROR after a preemption

Since #992, a preempted Stage 12 segment runner cancels the segment calls it started (stage12_runtime/fanout.py), and a preempted Stage 12 coordinator with a report run cancels its unfinished simulation calls (report_coordination.py, stop_children()); a segmented US simulation cancelled that way cancels its own started segments in turn. Modal 1.4.2 ends a running sync call it cancels by raising modal.exception.InputCancellation in it, which is a BaseException but not a KeyboardInterrupt, so 3.0.7 records each cancelled call's operation with outcome error at ERROR. One coordinator preemption can therefore write an ERROR record for each simulation and each of its running segments. This PR doesn't change that; the observability skill and the changelog fragment say so. Recording a cancelled call differently needs a change in policyengine-observability or in how these workers end a cancelled call; no issue is open for it yet.

What else changes from 3.0.4 to 3.0.7

• 3.0.5: a record written while a FastAPI endpoint runs (record_exception, event, log) carries the matched route template as http.route, such as /jobs/{job_id}, instead of <unmatched>. This affects the gateway, the entrypoint and the executor's local FastAPI app. The changelog fragment says so. The Public API's daily telemetry report excludes health-check routes by http.route (.github/scripts/telemetry_error_issues/bigquery.py, HEALTH_CHECK_ROUTES), so an ERROR record written inside a health-check handler, which carried <unmatched> and was counted, now carries the health-check route and is left out of the report.
• 3.0.6: only a newer certifi in the observability repository's own uv.lock; the package's requirements are unchanged.
• 3.0.7, besides the operation outcome:
  • The request and operation count and duration metrics get interrupted as a new value of their outcome attribute.
  • A span that ends in KeyboardInterrupt keeps an unset status (3.0.4 set it to error).
  • The FastAPI adapter passes a KeyboardInterrupt to the request record. 3.0.4 recorded such a request from its status code alone, as success at INFO when no status had been sent; it is now interrupted at WARNING. No preemption in these services reaches that code: the gateway is a Modal ASGI app, and Modal cancels an async app's running task on the first SIGINT (modal/_runtime/user_code_event_loop.py), so a preempted gateway request ends in asyncio.CancelledError, which 3.0.7 handles as 3.0.4 did. The changelog fragment leaves this out for that reason.
  • A span is closed even when adding its exception event or status fails. Failures to annotate a span are reported in diagnostics as otel.span_annotate, and otel.span_end counts only failures to close a span. Nothing in this repository reads either diagnostic name.
• The changelog fragment describes the 3.0.5 and 3.0.7 changes.

Overlap with #1019

Open draft #1019 relocks the same six lock files to 3.0.6 and raises the same floors to >=3.0.6. Whichever of the two merges second needs a relock (uv lock --upgrade-package policyengine-observability in each of the six projects) and its floors and docs adjusted to the other's; resolving those conflicts must keep >=3.0.7, or the preemption fix can be dropped by a later relock. If this one merges first, #1019's lock and floor changes are already on main, and it reduces to its route tests, its documentation paragraph and its changelog fragment.

Deployment

The gateway's image and the executor's simulation, Stage 12 and Stage 12 precompute images install from these locks with uv_sync(frozen=True), and the entrypoint's Dockerfile runs uv sync --frozen, so 3.0.7 reaches all of them on the next release.

A release deploys only the versioned executor app for the current policyengine.py version, policyengine-simulation-py<version> (simulation-deploy.reusable.yml, step "Select versioned executor app"). Executor apps for older policyengine.py versions stay deployed with 3.0.4 and still serve requests that pin those versions, so their preemptions are still recorded at ERROR until they are redeployed. Unpinned requests reach the current release. 3.0.5 to 3.0.7 read no new environment variable; the variables the observability runtime reads (OBSERVABILITY_*, OTEL_*, POLICYENGINE_OTEL_*) are the same as in 3.0.4, and this PR changes none of their providers or wiring.

Testing

Red then green. The first commit, 0f5fde26, adds the tests; they fail on 3.0.4 with assert 'ERROR' == 'WARNING' (and [('ERROR', 'error')] == [('WARNING', 'interrupted')]). With this PR's locks they pass (6 passed):

• projects/policyengine-simulation-executor:
  • tests/test_job_progress.py::TestModalRunSimulation::test__preempted_job_is_recorded_at_warning: run_simulation preempted during the calculation.
  • tests/test_job_progress.py::test_preempted_segment_is_recorded_at_warning: run_simulation_segment preempted.
  • tests/test_stage12_modal_app.py::test_stage12_functions_record_a_preemption_at_warning, for run_single_simulation_us, run_single_simulation_uk, run_single_simulation_segment_us and coordinate_report.
  • Each test runs the Modal function body inside a real observability runtime (fixture recorded_observability_runtime in tests/conftest.py, which keeps every structured record and, like the deployed workers' runtime, also records standard-library log records), raises KeyboardInterrupt inside it, checks the exception propagates, checks the operation.completed record has severity WARNING and outcome interrupted, and checks no ERROR record was written.
• The six tests also pass after merging main at 0576797f, which includes Record report runs and simulation runs from the v2 Modal coordinator and workers #992.
• uv lock --check --project <dir> passes for all six relocked projects after that merge.
• python -m scripts.rules pyright --sync <package> passes with no Pyright errors for all six on 193ff0a8 (before the second merge from main).
• Ruff 0.15.22 (CI's pinned release) check and format --check pass on the changed test files.
• BASE_SHA=origin/main .github/scripts/check-changelog-fragment.sh passes.
• I did not run the full test suites locally; CI runs them.

🤖 Generated with Claude Code