Skip to content

fix(subscriber): terminalize runs whose dispatch gives up before execution starts (#49604) - #41

Open
echoomegaprime wants to merge 1 commit into
mainfrom
fix/dispatch-exhaustion-orphans-49604
Open

echoomegaprime wants to merge 1 commit into
mainfrom
fix/dispatch-exhaustion-orphans-49604

Conversation

@echoomegaprime

Copy link
Copy Markdown
Owner

Echo fleet #49604. Not deployed. CertForge goes out through its own governed release.

Problem

When the last dispatch attempt errored (for example subscriber_worker_claim_denied) while the run's reservation was still BOUND, finish_dispatch() marked only the dispatch FAILED. The run itself was never transitioned. As a result:

  • the run stayed QUEUED forever;
  • the metered run was never compensated;
  • every "queue idle" check stayed blocked, including echo-swarm-recert (#46741), which has been exiting with STOP CertForge queue busy while the swarm's production certificate has expired.

On FORGE, 7 runs from 2026-10-09 to 10-30 are stuck exactly this way: dispatch FAILED after 3 attempts, reservation BOUND, execution never started, no compensation.

Fix

  • New helper _terminalize_unstarted_run(): gives the same outcome as the existing lease-expiry exhaustion path. The run moves to INFRASTRUCTURE_FAILURE with outcome INFRA_FAILED, the reservation is RELEASED, and the meter is compensated using the same dispatch-crash key, so a run can never be compensated twice.
  • finish_dispatch(): calls it for an error on the last attempt while the reservation is BOUND. The other branches are unchanged.
  • _recover_expired_dispatches(), which runs on every claim_dispatch: reconciles runs the old code already orphaned (reason dispatch_failed_run_orphaned, with an audit event). It is idempotent.

The existing lease-expiry block is untouched.

Tests

  • 4 new tests in tests/test_dispatch_orphaned_runs.py. On unmodified main, 3 of them fail, with the run left QUEUED.
  • The 7 subscriber and dispatch test modules pass, 107/107, and 6/6 mutants are killed.
  • A read-only dry run of the reconciler's selection against the live FORGE DB matches exactly the 7 orphans and nothing else.

Release notes

  • The deployed release d8ad9fd8 has the same subscriber.py as main (sha 482a0b4b…), so this applies cleanly.
  • On the first claim_dispatch after deploy, the 7 orphans move to INFRASTRUCTURE_FAILURE and each one's metered run is compensated once.
  • After that, re-run echo-swarm-recert.
  • PR subscriber: do not mask a failed BEGIN with 'cannot rollback' (fleet #46912) #40 (#46912) is also still waiting for a release.

🤖 Generated with Claude Code

https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec

…ution starts (#49604)

finish_dispatch() marked a dispatch FAILED when its last attempt
errored (for example subscriber_worker_claim_denied) with the run's
reservation still BOUND, but never transitioned the run. The run stayed
QUEUED forever, the metered run was never compensated, and every
"queue idle" check (swarm re-certification, etc.) stayed blocked. On
FORGE seven runs (2026-10-09..10-30) were stuck exactly this way.

- New _terminalize_unstarted_run(): QUEUED -> INFRASTRUCTURE_FAILURE,
  run_outcome INFRA_FAILED, BOUND reservation RELEASED, meter
  compensated with the same 'dispatch-crash' key as lease-expiry
  exhaustion, so a run is never compensated twice.
- finish_dispatch(): use it for an error on the last attempt with a
  BOUND reservation (QUEUED run). Retries, completion and the
  RELEASED-reservation path are unchanged.
- _recover_expired_dispatches() (run on every claim_dispatch): reconcile
  runs already orphaned by the old code (dispatch FAILED, reservation
  BOUND, run QUEUED) with reason dispatch_failed_run_orphaned and an
  audit event. Idempotent: terminalized runs no longer match.
- The existing lease-expiry exhaustion block is unchanged.

Tests: 4 new (tests/test_dispatch_orphaned_runs.py); the 7 subscriber
and dispatch test modules pass 107/107; 6/6 mutants killed. A read-only
dry run of the reconciler selection on the live FORGE DB matches exactly
the 7 orphans and nothing else.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec
echo-fleet-builder Bot pushed a commit that referenced this pull request Nov 3, 2026
…ompensation

Port PR #40 and PR #41 onto deployed d8ad9fd without unrelated main changes.
Retain historical crash credits by tenant/run identity and use a separate
run-compensation namespace for new credits. Add atomicity, concurrency,
legacy-credit, original-period, collision-order, and migration rollback tests.

Verified: 705 passed, 1 Docker-dependent skip; P1/P6/P7 acceptance green;
87 main integration tests passed; 7/7 isolated mutation checks killed.
No deployment, production state repair, or live compensation authorized.

Copy link
Copy Markdown
Owner Author

Integration check — 2026-11-10, isolated FORGE worktree

PR #40 (subscriber failed-BEGIN rollback guard) and PR #41 (terminalize exhausted QUEUED dispatches) were applied sequentially onto the current origin/main commit 6515544d0083edde1eefba8bd89ab3d5c95ccf4b in a separate worktree /tmp/cf-pr40-pr41-compat-lmpqxtw9. The source checkout with unrelated tracked edits was not modified.

Both git apply --check passes succeeded and both patches applied without conflicts. The focused tests tests/test_subscriber_migration_rollback.py and tests/test_dispatch_orphaned_runs.py ran together: 7 passed in 4.23s. PR #40 patch SHA-256 55c8377de5d2e3672291fcab777b29766526f6aa2799d61838dd8236451824c1; PR #41 patch SHA-256 1b30ca42dd810a32ebe5deda92592578a0f0e7ea8a1a53fa46066e761d301103.

A broader subscriber-governance test invocation did not return a reliable tool result, so a full 107-test regression claim is NOT made. Current exact-head hosted CI, release certification, and production promotion were not run; the PRs remain unmerged.

Current Certification Forge readback: most recent sampled release verdict is signed, signature-valid and NOT_READY for mandatory_rule_failed:production_e2e / production_e2e_attestation_missing. Successful patch compatibility does not satisfy that release gate.

Next: rerun bounded subscriber suite with inspectable result; establish exact merged candidate and passing CI/security; use CertForge signed PRODUCTION_READY and rollback evidence before any merge/promotion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants