[anth-volk]

Fixes #1058

What this does

When the Simulation Entrypoint creates a household job, report run or standalone simulation run but stops before starting its Modal call, nothing records a call for it. Until now the stuck-run sweep failed such a run with run_not_started at 10 minutes. With this change, a restart check starts the run's call once more, and the sweep fails it only if that restart misses too.

Where it runs. It is a second scheduled function, restart_unstarted_runs, in the stuck-run sweep's Modal app (policyengine-simulation-run-sweep). It runs every two minutes as the Modal worker role, from the sweep's Modal secret policyengine-modal-worker-database. The restart code itself is the simulation service's (policyengine_simulation_entry.run_restarts, run_restart_calls, run_restart_check), on policyengine_simulation_persistence (run_restart_records, run_restart_store). The image adds both as local sources.

What one check does:

1. Lists, oldest first and at most 20, the rows created 3 to 10 minutes ago that are still pending with no modal_call_id and whose updated_at still equals created_at:

   • household jobs;
   • report runs outside a budget window whose baseline is an economy simulation;
   • runs of economy simulations that no report run names (standalone runs).

   A report's own simulation runs are its coordinator's to start, so they are never listed.

2. Reads back, on a thread, what the submission built the call from, each side's values read against the run's own catalog version with the submission's own reader. Then it rebuilds the call with the submission's own builders (report_run_calculation, simulation_run_calculation, prepare_household_job). It uses the manifest's worker for the .py version the row records, even when that worker is no longer the default.

   • The rebuilt payload equals the original. The context differs only in its request ID (run-restart-<uuid>).
   • A household job whose rebuilt job would record anything else is not restarted: for example, a changed default year or catalog version.

3. Sets the row's updated_at (with clock_timestamp()), so the row is restarted at most once, even by two checks at once. Then it starts the call and records its ID with the submission's own conditional write.

4. Counts the restart as done when the row then records the check's call. A row recording another call was claimed by that call first.

After three minutes a check marks no more rows and leaves the rest for the next check. A marked row then needs only its call's start and a few bounded statements, so the check ends within the function's five-minute timeout, and every row it marks gets its call recorded before the sweep's 15 minutes.

When a restart can't go ahead:

• A row whose call can't be rebuilt is failed at once with run_not_started: …. That covers a worker that left the manifest, stored rows that changed, and a policy that is gone. A report run is failed together with its simulation runs that are still pending with no call, as its submission would fail it.
• So is a row whose restarted call Modal refuses, or doesn't confirm within ten seconds.
• A row whose rows or manifest can't be read right now is left unmarked, and the next check tries it again. So is a row whose rebuild hits an unexpected error, which is logged with its traceback (v2_run_restart_unexpected_error); the check goes on to the next row.
• A late original call claims a row only while the row records no call or its own. Once the restart records its call, the late call's claim is refused. If the late call claims first, the restarted call's claim is refused instead.

The sweep now waits 15 minutes (DEFAULT_MINIMUM_AGE, was 10) before failing a pending run with no call. That is after the restart window, so a run whose restart started is judged by its call, and a second miss is still failed with run_not_started.

Telemetry (how often runs need a restart):

• Each check is a v2_run_restart_check operation. The counts are set on it as restart_listed, restart_restarted, restart_failed, and so on.
• Each row a check takes up is a v2_run_restart operation with run_kind, run_id and run_age_seconds. A deferred row is taken up again by the next check, so the runs whose call never started are the distinct run_id values.
• Each outcome is set on that operation (restart_outcome, restart_error_type). It is also logged as a v2_run_restart_outcome event, at WARNING when the row was failed, couldn't be read or couldn't be recorded.
• The function's return value is the same counts, which Modal's dashboard shows per run.

Smaller changes:

• ManifestEconomyWorkers and ManifestHouseholdWorkers gain worker_for_version.
• The entry package's __init__ loads create_app on first use. Before, importing any of its modules imported the FastAPI app, whose PyJWT and cryptography the sweep's image doesn't install.
• The executor's dev group gains the entry and persistence packages. The deploy job's uv sync installs them for modal deploy, which adds them to the sweep image as local sources. The executor's own Docker image is built without that group, so it doesn't need them. The relock adds only those two.

What waits for other pull requests

• Calculate budget windows on the detailed-save path, five years at a time #1054 (C1 · 13, budget windows), merged while this branch was in review; main is merged in next. Report runs with a budget_window_run_id are not restarted: a year still waiting in its window is pending with no call by design. Restarting a year whose start an earlier year's coordinator lost waits until Calculate budget windows on the detailed-save path, five years at a time #1054 merges. Calculate budget windows on the detailed-save path, five years at a time #1054 also changes run_sweep.py; this branch changes only DEFAULT_MINIMUM_AGE there. Calculate budget windows on the detailed-save path, five years at a time #1054's idle-window settling uses the same minimum age, so it moves to 15 minutes too.
• C1 · 15 (household reports). Runs of household simulations, and report runs whose baseline is one, are left out until that pull request merges and the restart learns to rebuild their call.
• C1 · 17 (re-runs). Re-runs need no extra code if each new run records its own year, catalog version, dataset and region, and records its call on the report run or standalone run, as POST /reports and POST /simulations do. That is being confirmed with that session.

Once each merges, main is merged into this branch and the restart covers what it adds.

Deployment

The deploy step Deploy the stuck-run sweep and the restart check in publish_stage12_v2_default deploys the app, as before. It runs only from main, after every check. Each variable the changed code or workflow reads or declares:

Variable	Provider	Wiring	Confirmed
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	GitHub Actions secrets	The step's env: (unchanged), for modal deploy.	Yes, unchanged
SIMULATION_V2_MODAL_ENVIRONMENT	The reusable workflow's modal_environment input (staging or main, from simulation-deploy.yml)	The step's env:. run_sweep_app.py bakes it into the image's environment at deploy time. The restart function reads it, then maps it to the deployment label (staging or production) that a call's context and artifact prefix name.	Yes. The input is used by the same step's command.
OBSERVABILITY_*, OTEL_EXPORTER_OTLP_ENDPOINT, POLICYENGINE_OTEL_GOOGLE_AUDIENCE	GitHub Actions repository variables (vars.OBSERVABILITY_*)	The step's env:, the same eight entries as the v2 workers' deploy step. modal_image_environment() bakes them into the image.	Yes. The variables exist at repository level.
MODAL_WORKER_DATABASE_URL	Modal secret policyengine-modal-worker-database	Attached to both scheduled functions, with required_keys.	Yes. The sweep already requires it in staging and main.
MODAL_IDENTITY_TOKEN, MODAL_TASK_ID, MODAL_REGION	Modal, at run time	Read by the observability runtime.	Provided by Modal

Telemetry export needs an operator step (not done). The modal-api-v1 workload identity provider (project policyengine-observability) admits only these app names:

• policyengine-simulation-gateway;
• names matching policyengine-simulation-py…;
• names matching policyengine-simulation-v2-py….

So the sweep's app can't get Google credentials. Until its condition also admits policyengine-simulation-run-sweep, the restart check's records reach Modal's own logs but not Cloud Logging or the collector.

Tests

All run by node ID; CI runs the rest.

• Persistence: the 18 tests of test_run_restart_records.py, against a PostgreSQL 17 database built by the Public API master's v2 chain (429d348ae), with its privileges reconciled. They run as the Modal worker role.

• Entry:

  • the 26 tests (27 cases) of test_run_restarts.py;
  • the 7 tests (12 cases) of test_run_restart_check.py;
  • the 3 new tests each in test_economy_workers.py and test_household_workers.py.

• Executor:

  • the changed and new tests of test_stage12_run_sweep_app.py;
  • test_stage12_run_sweep.py::test_by_default_unstarted_runs_wait_until_the_restart_check_has_passed;
  • test_executor_image_dependencies.py, which checks the Docker image's build holds every main dependency from a local path;
  • test_run_restart_image.py, which imports the restart check with only the sweep image's packages, and requires every module it loads to come from the standard library, a local source or a distribution the image installs.

  A dry run of the Docker image's uv sync --frozen --no-dev, in a copy holding only its build directories, also succeeds.

• Observability: test_stages.py::test_v2_run_restart_plan_measures_each_check_and_each_restart.

Review

Five read-only review rounds, each by a reviewer following the review-code skill:

• Round 1 found three bugs, each fixed with a failing test first:
  • the executor's Docker image no longer built, because the entry package was a main dependency outside the image's build;
  • a report run whose rows couldn't be read was failed without its simulation runs;
  • a restart whose own call claimed the row before the record was counted as another call's.
• Round 2 found no bugs. Its quality findings were fixed:
  • the restart window is narrowed to 3–10 minutes, for a margin before the sweep;
  • an unexpected rebuild error defers only its own row;
  • the telemetry docs are corrected;
  • an image import test is added.
• Round 3 found no bugs. Its quality findings were fixed: the image test checks every module file the import loads, and unexpected errors are logged with their traceback.
• Round 4, on round 3's fixes, found one bug, fixed test-first: the unexpected-error log's run kind and ID didn't reach the observability runtime's handler.
• Round 5, on round 4's fixes, found no bugs. Its findings were wording only.

🤖 Generated with Claude Code