Repository navigation
fix(subscriber): terminalize runs whose dispatch gives up before execution starts (#49604) - #41
echoomegaprime wants to merge 1 commit into
Conversation
…ution starts (#49604) finish_dispatch() marked a dispatch FAILED when its last attempt errored (for example subscriber_worker_claim_denied) with the run's reservation still BOUND, but never transitioned the run. The run stayed QUEUED forever, the metered run was never compensated, and every "queue idle" check (swarm re-certification, etc.) stayed blocked. On FORGE seven runs (2026-10-09..10-30) were stuck exactly this way. - New _terminalize_unstarted_run(): QUEUED -> INFRASTRUCTURE_FAILURE, run_outcome INFRA_FAILED, BOUND reservation RELEASED, meter compensated with the same 'dispatch-crash' key as lease-expiry exhaustion, so a run is never compensated twice. - finish_dispatch(): use it for an error on the last attempt with a BOUND reservation (QUEUED run). Retries, completion and the RELEASED-reservation path are unchanged. - _recover_expired_dispatches() (run on every claim_dispatch): reconcile runs already orphaned by the old code (dispatch FAILED, reservation BOUND, run QUEUED) with reason dispatch_failed_run_orphaned and an audit event. Idempotent: terminalized runs no longer match. - The existing lease-expiry exhaustion block is unchanged. Tests: 4 new (tests/test_dispatch_orphaned_runs.py); the 7 subscriber and dispatch test modules pass 107/107; 6/6 mutants killed. A read-only dry run of the reconciler selection on the live FORGE DB matches exactly the 7 orphans and nothing else. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec
…ompensation Port PR #40 and PR #41 onto deployed d8ad9fd without unrelated main changes. Retain historical crash credits by tenant/run identity and use a separate run-compensation namespace for new credits. Add atomicity, concurrency, legacy-credit, original-period, collision-order, and migration rollback tests. Verified: 705 passed, 1 Docker-dependent skip; P1/P6/P7 acceptance green; 87 main integration tests passed; 7/7 isolated mutation checks killed. No deployment, production state repair, or live compensation authorized.
Integration check — 2026-11-10, isolated FORGE worktreePR #40 ( Both A broader subscriber-governance test invocation did not return a reliable tool result, so a full 107-test regression claim is NOT made. Current exact-head hosted CI, release certification, and production promotion were not run; the PRs remain unmerged. Current Certification Forge readback: most recent sampled release verdict is signed, signature-valid and NOT_READY for Next: rerun bounded subscriber suite with inspectable result; establish exact merged candidate and passing CI/security; use CertForge signed |
Echo fleet #49604. Not deployed. CertForge goes out through its own governed release.
Problem
When the last dispatch attempt errored (for example
subscriber_worker_claim_denied) while the run's reservation was stillBOUND,finish_dispatch()marked only the dispatchFAILED. The run itself was never transitioned. As a result:QUEUEDforever;STOP CertForge queue busywhile the swarm's production certificate has expired.On FORGE, 7 runs from 2026-10-09 to 10-30 are stuck exactly this way: dispatch
FAILEDafter 3 attempts, reservationBOUND, execution never started, no compensation.Fix
_terminalize_unstarted_run(): gives the same outcome as the existing lease-expiry exhaustion path. The run moves toINFRASTRUCTURE_FAILUREwith outcomeINFRA_FAILED, the reservation isRELEASED, and the meter is compensated using the samedispatch-crashkey, so a run can never be compensated twice.finish_dispatch(): calls it for an error on the last attempt while the reservation isBOUND. The other branches are unchanged._recover_expired_dispatches(), which runs on everyclaim_dispatch: reconciles runs the old code already orphaned (reasondispatch_failed_run_orphaned, with an audit event). It is idempotent.The existing lease-expiry block is untouched.
Tests
tests/test_dispatch_orphaned_runs.py. On unmodifiedmain, 3 of them fail, with the run leftQUEUED.Release notes
d8ad9fd8has the samesubscriber.pyasmain(sha482a0b4b…), so this applies cleanly.claim_dispatchafter deploy, the 7 orphans move toINFRASTRUCTURE_FAILUREand each one's metered run is compensated once.🤖 Generated with Claude Code
https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec