US production: /us/metadata returns 500 and state-region simulations fail with connection reset at the simulation gateway

[MaxGhenis]

Summary

The US v1 production stack is partially down, and monitoring can't see it because Better Stack only probes the API root (which returns 200):

1. GET /us/metadata returns HTTP 500 — observed on 2026-08-06, 2026-08-15, and 2026-08-21 (every check). /uk/metadata and / return 200. The app's US parameter pages depend on this endpoint.
2. US state-region society-wide simulations fail — GET /us/economy/{policy}/over/2?region=state/or&time_period=2027 returns {"status": "error", "message": "[Errno 104] Connection reset by peer", "result": null}, consistently on retry. Previously cached state results now error too (the cached result is gone and the recompute fails).
3. National runs hang — region=us requests sit in "computing" for 10+ minutes (normal completion is a few minutes).

As of 2026-08-21 18:00 UTC, every executor app in the Modal main environment (policyengine-simulation-py4-18-*, py4-20-3, py4-22-0) and policyengine-simulation-gateway show deployed with 0 running tasks while the API reports runs computing — requests are not starting on Modal. The reset-by-peer surfaces at the API → gateway → executor hop for the route production uses (resolved_app_name: policyengine-simulation-py4-20-3 on completed runs from before the failure).

Timeline (all observations read-only, from outside)

• 2026-07-17: state runs healthy (e.g. policy 98001 state/or 2027 → −$25.2M, stamped pe-us 1.764.6 / populace-us-2024-buildi-sparse… / policyengine-simulation-py4-20-3).
• 2026-08-06: /us/metadata 500; state runs still complete with the identical stamp.
• 2026-08-15: /us/metadata 500; state runs still complete.
• 2026-08-21: /us/metadata 500; state runs fail with connection reset; national hangs.

Better Stack logged only two brief auto-resolved root incidents (8/7 09:14–09:19, 8/14 04:47–04:52) in the same window.

Repro

curl -s -o /dev/null -w "%{http_code}\n" https://api.policyengine.org/us/metadata      # 500
curl -s -o /dev/null -w "%{http_code}\n" https://api.policyengine.org/uk/metadata      # 200
curl -s "https://api.policyengine.org/us/economy/98001/over/2?region=state/or&time_period=2027"
# {"status": "error", "message": "[Errno 104] Connection reset by peer", "result": null}

Possibly related

• The metadata failure window starts after the v1 persistence migration to SQLAlchemy/Alembic (Migrate API v1 persistence to SQLAlchemy and Alembic #3788, 8/12) and Cloud Run revision changes in sim-api (Update PolicyEngine US to 0.416.2 #657, 7/28) — correlation only; I have not looked at logs.
• Simulation API runs that complete with 500 errors cause the API to respond with 200 and a result of None #2430 (simulation-API 500s surface as status: ok + result: null) describes the adjacent failure mode; this one at least surfaces as status: error.
• Suggest adding /us/metadata and a state-region simulation smoke to Better Stack — the root probe can't catch either.

User-facing impact: every state-level analysis on policyengine.org (US) currently fails; external partners sharing state reform links hit errors.

[MaxGhenis]

Status update (2026-08-22 ~16:20 UTC): production has recovered on its own; root causes below

Read-only diagnosis, no production changes made. Everything below is from live probes, code on master, Modal CLI reads, and GitHub Actions logs.

Current state (verified 2026-08-22 15:45–16:25 UTC)

• GET /us/economy/98001/over/2?region=state/or&time_period=2027 → status: ok, −$25,231,108.36, stamped pe-us 1.764.6 / buildi / policyengine-simulation-py4-20-3 (the stored result).
• A fresh state compute (region=state/wa, request id probe-3797-fresh2) went end-to-end: API → simulation entrypoint → Modal gateway (POST /simulate/economy/comparison → 200, job fc-01M0N47333QJDHCYXN0SQGT2S9, resolved policyengine-simulation-py4-20-3) and completed in ~3 min.
• region=us returns status: ok (stored result); a fresh UK national run also completed (2.89.2).
• /simulation-gateway-check → {"status":"healthy","simulation_gateway":"healthy"}; the entrypoint (https://simulation.api.policyengine.org/health, revision policyengine-simulation-entry-00011-vom) is healthy.
• GET /us/metadata still returns 500 for clients that do not negotiate gzip (see below — this one is fully explained and has a fix PR).

Symptom 1: /us/metadata 500 — root cause confirmed, fix in #3798

The 500 is generated by the Google Frontend, not the app (empty text/html body, no x-policyengine-* headers). /us/metadata serializes to 71,532,620 bytes and both metadata routes send it with Content-Length. Cloud Run drops HTTP/1 responses above 32 MiB unless they stream (quotas), so:

• curl without --compressed (and any non-gzip monitor) → 500.
• Browsers, python requests/httpx, and anything sending Accept-Encoding: gzip → 200 (GZipMiddleware compresses to ~10 MB, under the cap). The web app was never affected.

This started when the public host moved from App Engine (nginx sidecar handled gzip) to Cloud Run — which brackets the first observed failure on 8/6. It predates and is unrelated to #3793. Fix: #3798 streams bodies ≥ 20 MiB (chunked, no Content-Length) in both the native FastAPI route and the Flask fallback; smaller responses are byte-identical to today.

Symptom 2/3: state-run "[Errno 104] Connection reset by peer" + national hangs — mechanism confirmed, trigger needs Cloud Run logs

The mechanism, verified in code on master:

1. Economy GETs that find a computing reform-impact row poll the simulation entrypoint: _handle_computing_impact → SimulationEntrypoint.get_execution_by_id → httpx GET {SIMULATION_ENTRYPOINT_URL}/jobs/{id} (simulation_entrypoint.py:319–366). Fresh submissions hit the same base URL.
2. On httpx.RequestError the client logs "Simulation entrypoint request error polling job {id}: ..." at severity ERROR and re-raises.
3. Nothing catches it until Flask's generic handler (error_routes.py:47–50), which returns exactly {"status": "error", "message": str(e), "result": null}. [Errno 104] Connection reset by peer is the stringified transport error from the API→entrypoint hop.

So during the outage window the API's outbound connections to https://simulation.api.policyengine.org were being reset. That fits every observation: cached-looking results erroring (they had live computing rows to poll), fresh submits failing identically, national runs stuck in computing (jobs never started — every executor showed 0 running tasks and the gateway received nothing), and "consistent on retry". It also fits the recovery without any deploy: policyengine-api last deployed 8/19 18:24 UTC, sim-api main hasn't changed since 8/4, the gateway's last Modal deploy is v78 (8/4), and the routing dict (4.20.3 → policyengine-simulation-py4-20-3) is intact — the entrypoint hop simply started working again, stuck rows got re-polled/refreshed, and results re-completed.

What I could not determine without GCP access (my gcloud auth is expired): why the entrypoint reset connections between 8/15 and 8/21 and what changed before 8/22. The answers are in Cloud Run logs; queries that will settle it:

# API side: the ERROR-severity polling failures (exact window + count)
gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="policyengine-api" severity>=ERROR timestamp>="2026-08-14T00:00:00Z" "Simulation entrypoint request error"' --project policyengine-api --limit 50 --format json

# Entrypoint side: request outcomes, instance starts/crashes, throttling
gcloud logging read 'resource.type="cloud_run_revision" resource.labels.service_name="policyengine-simulation-entry" timestamp>="2026-08-14T00:00:00Z" severity>=WARNING' --project policyengine-simulation-entry --limit 100 --format json

(The entrypoint runs in project policyengine-simulation-entry, us-central1, 1 vCPU / 512 Mi / concurrency 80 / 60 s timeout, behind its own load balancer.)

Monitoring gap

Better Stack probes only /, which stayed 200 throughout. Proposed monitors:

1. GET https://api.policyengine.org/us/metadata without Accept-Encoding, expect 200 (catches the size regression; will pass once Stream metadata responses above Cloud Run's non-streamed size cap #3798 deploys).
2. GET https://api.policyengine.org/simulation-gateway-check, expect 200 (catches the entrypoint hop dying — this would have caught the 8/21 outage).
3. Optionally a state-region economy smoke on a stored result.

Release train (separate from the incident but discovered en route)

The bundle-bump PRs are stuck on the "Ensure PolicyEngine bundle is supported by simulation API" guard: policyengine.py 5.0.x has no deployed executor, and the sim-api updater run for 5.0.3 failed because the executor pins policyengine-core==3.30.0/policyengine-uk==2.89.2 while pe.py 5.0.3 moved to core 3.30.1 / uk 2.90.2. sim-api #664 (5.0.2) + api #3791 (5.0.2) are green and mergeable, and 5.0.2 carries model pins identical to the serving 4.20.3 (us 1.764.6 / uk 2.89.2 / core 3.30.0) — a pure infra bump. Sequencing is being handled separately.

Diagnosis run by Claude (read-only) for Max; happy to hand over logs/details.

[MaxGhenis]

Resolution status (2026-08-23 08:55 UTC)

• /us/metadata 500 — fixed in production. Stream metadata responses above Cloud Run's non-streamed size cap #3798 merged (master 19da6ca) and deployed; GET /us/metadata without Accept-Encoding now returns 200 (71,532,620 bytes streamed, 4.5 s, x-policyengine-backend: cloud_run); the gzip path is unchanged (200, ~9.96 MB).
• State-region and national runs — healthy. 98001 over 2, state/or, 2027 → ok, −$25,231,108.36, pe-us 1.764.6 / policyengine-simulation-py4-20-3; region=us → ok; fresh computes verified end-to-end through the entrypoint and Modal gateway on 8/22. /simulation-gateway-check 200.
• Also shipped en route: Pin anthropic below 1.0 and guard the SDK call contract #3799 (pins anthropic<1 — the 1.0 SDK release on 8/20 had broken every PR's Test job and would have broken the next production image; deployed 8/22) and AI analysis calls a retired model (claude-sonnet-4-20250514 → 404 not_found_error) #3800 (AI-analysis model is retired; parked until the Axiom migration).

Still open from the original report:

1. Why the entrypoint hop reset connections 8/16–8/21 — needs the two Cloud Run log queries in the RCA comment above (API project + policyengine-simulation-entry).
2. Monitors — Better Stack checks for GET /us/metadata without gzip and GET /simulation-gateway-check (the second would have caught this outage).

Closing is reasonable now on the user-facing symptoms; leaving open only if you want this issue to track the two follow-ups.