Require policyengine-observability 3.0.7 so Modal preemptions are recorded at WARNING

changelog.d/record-preemptions-as-interrupted.fixed.md

@@ -0,0 +1,5 @@
+A Modal preemption of a simulation worker, segment worker or Stage 12 function is now recorded as an interrupted operation at WARNING instead of a failed one at ERROR, so the preempted call no longer opens telemetry error issues. Modal delivers a preemption to these synchronous functions as `KeyboardInterrupt` and runs the same input again; the observability library, the executor, the gateway and the entrypoint now require policyengine-observability 3.0.7, which records an operation or request that ends in `KeyboardInterrupt` with the outcome `interrupted` at WARNING and leaves it out of `policyengine.error.count`. The request and operation count and duration metrics get `interrupted` as a new value of their `outcome` attribute. An operation's span, or a span opened with `runtime.span`, that ends in `KeyboardInterrupt` is no longer set to error: its status stays unset and it gets the attribute `policyengine.outcome` set to `interrupted`. This applies to the executor app this release deploys; executor apps deployed for older policyengine.py versions keep the earlier behaviour until they are redeployed.
+
+A child call that a preempted Stage 12 segment runner or report coordinator cancels still writes an ERROR record, and so does each running segment of a segmented US simulation cancelled that way: Modal ends a cancelled call with `InputCancellation`, which is not a `KeyboardInterrupt`.
+
+The move from 3.0.4 also brings 3.0.5's change: a record written while a FastAPI endpoint runs, such as `exception.recorded`, carries the matched route template as `http.route` instead of `<unmatched>`.

docs/engineering/skills/observability.md

@@ -80,9 +80,9 @@ appropriate `StagePlan` and call `plan.name(Stage.VALUE)`. Do not add span name
 string literals in services or workers.
 
 Open a span with `runtime.span(plan.name(Stage.VALUE))`, and an operation with
-`runtime.operation(...)`. policyengine-observability 3.0.4 types both scopes
-correctly: their `__exit__` and `__aexit__` return `Literal[False]`, so Pyright
-knows an exception raised in the block propagates; see
+`runtime.operation(...)`. policyengine-observability 3.0.4 and later type both
+scopes correctly: their `__exit__` and `__aexit__` return `Literal[False]`, so
+Pyright knows an exception raised in the block propagates; see
 `docs/engineering/skills/type-checking.md`.
 
 When adding a run configuration or runtime stage:
@@ -122,6 +122,33 @@ Modal can run the same call again after a preemption, under the same call
 id. A later attempt continues the stored stage record, so the durations add
 up over attempts, and `job_attempts` counts them.
 
+### Preemptions
+
+Modal delivers a preemption to a running synchronous function, which every
+executor and Stage 12 Modal function is, as `KeyboardInterrupt`, then runs
+the same input again. In an async app, such as the gateway's ASGI app, Modal
+cancels the running task instead, so a request there ends in
+`asyncio.CancelledError` and what follows doesn't apply to it.
+policyengine-observability 3.0.7 and later
+record an operation or request that ends in `KeyboardInterrupt` with the
+outcome `interrupted` at WARNING and leave it out of the error count metric;
+a span that ends in it keeps an unset status and gets the attribute
+`policyengine.outcome` set to `interrupted`. The exception propagates
+unchanged, so the preempted call's own records don't reach the daily
+telemetry error report. Let `KeyboardInterrupt` leave the operation block as
+it is: code that catches it and raises another exception instead reports the
+attempt as a failure at ERROR. `except Exception` doesn't catch it.
+
+Modal ends a call it cancels with `InputCancellation` instead, which is a
+`BaseException` but not a `KeyboardInterrupt`, and policyengine-observability
+records that operation with the outcome `error` at ERROR. A preempted Stage 12
+segment runner cancels the segment calls it started, and a preempted Stage 12
+coordinator with a report run cancels its unfinished simulation calls; a
+segmented US simulation cancelled that way cancels its own started segment
+calls in turn. Each of those calls that was running writes an ERROR record,
+so one coordinator preemption can write one for each simulation and each of
+its running segments.
+
 ## Metric resource identity
 
 Every process that emits OpenTelemetry metrics must have a unique

docs/engineering/skills/type-checking.md

@@ -174,13 +174,14 @@ section is still in the registry fails the guard on every pull request.
 - If an error shows a real bug, first write a test that demonstrates it, then
   fix it.
 - Open observability spans and operations with `runtime.span(...)` and
-  `runtime.operation(...)`. policyengine-observability 3.0.4 types them
-  correctly: their `__exit__` and `__aexit__` return `Literal[False]`, so
+  `runtime.operation(...)`. policyengine-observability 3.0.4 and later type
+  them correctly: their `__exit__` and `__aexit__` return `Literal[False]`, so
   Pyright knows they never suppress an exception and doesn't report a variable
   the block assigns as possibly unbound after it. Every package that depends
-  on policyengine-observability requires 3.0.4 or later, and
+  on policyengine-observability requires 3.0.7 or later, which also records
+  a preemption at WARNING (see `observability.md`), and
   `libs/policyengine-simulation-observability/tests/test_scope_exit_typing.py`
-  fails if an older release is installed.
+  fails if a release older than 3.0.4 is installed.
 - Before reading a simulation's output frames, narrow
   `<simulation>.output_dataset.data` with the `EntityFrames` protocol in
   `policyengine_simulation_executor.output_frames`, and keep reading through

libs/policyengine-simulation-observability/pyproject.toml

@@ -10,7 +10,7 @@ requires-python = ">=3.13"
 dependencies = [
     "fastapi>=0.115.0",
     "pydantic>=2.0",
-    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.4,<4",
+    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.7,<4",
 ]
 
 [project.optional-dependencies]

projects/policyengine-simulation-entry/pyproject.toml

@@ -16,7 +16,7 @@ dependencies = [
     "fastapi>=0.115.0,<1",
     "httpx>=0.28.0,<1",
     "modal>=1.4,<2",
-    "policyengine-observability[fastapi,google,httpx,otlp-grpc]>=3.0.4,<4",
+    "policyengine-observability[fastapi,google,httpx,otlp-grpc]>=3.0.7,<4",
     "policyengine-simulation-contract",
     "policyengine-simulation-observability",
     "policyengine-simulation-persistence",

projects/policyengine-simulation-executor/pyproject.toml

@@ -24,7 +24,7 @@ dependencies = [
     "policyengine[models]==6.16.0",
     "tables>=3.10.2",
     "modal>=0.73.0",
-    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.4,<4",
+    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.7,<4",
     # Imported directly by the artifact store client; do not rely on the
     # transitive pin via policyengine.
     "google-cloud-storage>=2",
@@ -52,7 +52,7 @@ modal-simulation-image = [
     "policyengine[models]==6.16.0",
     "fastapi>=0.115.0",
     "tables>=3.10.2",
-    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.4,<4",
+    "policyengine-observability[fastapi,google,otlp-grpc]>=3.0.7,<4",
     # The artifact fetch layer and store client read GCS. The lib is also a
     # transitive dependency of policyengine, but the image must not depend
     # on an upstream package keeping it.

projects/policyengine-simulation-executor/tests/conftest.py

@@ -73,6 +73,58 @@ def real_observability_runtime(monkeypatch):
     runtime.shutdown()
 
 
+@pytest.fixture
+def recorded_observability_runtime():
+    """Build an observability runtime that keeps every record it writes.
+
+    Yields the runtime and the list its structured records are appended to,
+    in the order they were written. Like the deployed workers' runtime, it
+    also records standard-library log records, so a test sees an ERROR
+    record from ``logging`` too; ``runtime.shutdown()`` removes that handler.
+    OpenTelemetry is disabled and nothing is exported.
+    """
+    from policyengine_observability import (
+        CustomLogDestination,
+        DeploymentIdentity,
+        LoggingConfig,
+        ObservabilityConfig,
+        OTelConfig,
+        ServiceIdentity,
+        configure,
+    )
+
+    written_records: list[dict] = []
+
+    class ListRecordWriter:
+        def write(self, record: dict) -> None:
+            written_records.append(record)
+
+    runtime = configure(
+        ObservabilityConfig(
+            service=ServiceIdentity(
+                name="policyengine-simulation-executor-test",
+                namespace="policyengine.test",
+                version="test",
+                role="test",
+            ),
+            deployment=DeploymentIdentity(environment="test", platform="local"),
+            logging=LoggingConfig(
+                destinations=(
+                    CustomLogDestination(
+                        name="test-records",
+                        writer_factory=ListRecordWriter,
+                        delivery="inline",
+                    ),
+                ),
+                capture_standard_library=True,
+            ),
+            otel=OTelConfig(enabled=False),
+        )
+    )
+    yield runtime, written_records
+    runtime.shutdown()
+
+
 @pytest.fixture(autouse=True)
 def _forget_spm_runtime_caches():
     """Keep the SPM runtime memoization out of every other test's way.