Skip to content

Recalculate an incomplete Stage 12 cache file outside the output-plan check's except block - #1036

Merged
anth-volk merged 2 commits into
mainfrom
stage12-ensure-run-outside-handler
Nov 10, 2026
Merged

anth-volk merged 2 commits into
mainfrom
stage12-ensure-run-outside-handler

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Fixes #1035

Same-Repository Draft PR Check

  • This PR is a draft.
  • This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

  • This PR adds a changelog fragment, changelog.d/1035.fixed.md, whose type is fixed.

Summary

  • Stage12CachedSimulation.ensure() recalculated an incomplete Stage 12 cache file inside the except (TypeError, ValueError): block of its output-plan check, so any exception from run() or save(), including a Modal preemption's KeyboardInterrupt, carried the output-plan ValueError as __context__. The Public API's telemetry error report fingerprints an error by the first traceback in its stack, so these errors were grouped under the check instead of where they were raised.
  • The except block now only logs the same WARNING and notes that the file is incomplete; the recalculation (incomplete outcome, run(), save(), clearing .py's in-memory cache) runs after the try statement.
  • Everything else is unchanged: cache miss, missing output plan, cache hit, the WARNING message, the recorded outcomes, and exceptions other than TypeError or ValueError from the check, which still propagate.
  • This is the same pattern Run the simulation outside the saved-output lookup handler in Simulation.ensure() policyengine.py#746 fixes in .py's Simulation.ensure(), which the cache-miss path here still goes through until the service moves to a .py release with that fix.

Merge order: merge after #1033 (policyengine-observability 3.0.7), so the service records preemptions at WARNING before any error's fingerprint changes. Deploys run one at a time in merge order, so merging after #1033 is enough.

Testing

  • New tests in projects/policyengine-simulation-executor/tests/test_stage12_cache_runtime.py, run by node ID:
    • test_recalculation_error_carries_no_output_plan_error[run] and [save]: an error from run() or save() while recalculating an incomplete file is raised unchanged, with __context__ and __cause__ of None, and the outcome is incomplete.
    • test_preemption_during_recalculation_carries_no_output_plan_error: a KeyboardInterrupt from run() carries no __context__.
  • Red (commit 26782a5, tests only): all 3 failed, each with the output-plan ValueError as __context__. Green (with the change): all 3 pass.
  • Ruff 0.15.22 (the version CI pins): format --check and check pass on both changed files. Pyright was not run locally; CI runs it.
  • No other tests were run locally.

Review

Self-review by the authoring session; findings only, no review disposition.

  • Round 1 (a fresh reviewer given the diff and Recalculate an incomplete Stage 12 cache file outside the output-plan check's except block #1035): no bugs found. It checked every path through ensure() against main, confirmed the flag can't be unbound, and ran the 3 tests against main's module, where all 3 fail. Review stopped after round 1.
  • Noted, not changed here: src/modal/segmented_national.py:381-383 respawns an overdue child inside except TimeoutError:, so a spawn failure carries the poll's TimeoutError as context. Child run errors are not affected.

🤖 Generated with Claude Code

anth-volk and others added 2 commits November 10, 2026 18:13
Three tests fail on main: an error from run() or save(), and a KeyboardInterrupt from run(), raised while Stage12CachedSimulation.ensure() recalculates an incomplete cache file carry the output-plan ValueError as __context__.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage12CachedSimulation.ensure() now only notes in its except block that the loaded file failed the output plan, and runs and saves after the try statement, so a recalculation error or a preemption carries no chained output-plan error. Fixes #1035.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 10, 2026 14:45
@anth-volk
anth-volk merged commit 5a81542 into main Nov 10, 2026
28 checks passed
@anth-volk
anth-volk deleted the stage12-ensure-run-outside-handler branch November 10, 2026 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Recalculate an incomplete Stage 12 cache file outside the output-plan check's except block

1 participant