Repository navigation
subscriber: do not mask a failed BEGIN with 'cannot rollback' (fleet #46912) - #40
Open
echoomegaprime wants to merge 1 commit into
Open
echoomegaprime wants to merge 1 commit into
echoomegaprime wants to merge 1 commit into
Conversation
…46912) echo-certforge-dispatcher flapped at start-up on 2026-10-19 with sqlite3.OperationalError: cannot rollback - no transaction is active. When BEGIN IMMEDIATE itself fails in _migrate_legacy_platform_tables (for example 'database is locked' once the busy timeout expires while another process holds the write lock), the except branch ran ROLLBACK with no transaction open. That raised a second error, hid the real cause, and crashed the worker. The handler now rolls back only when connection.in_transaction is set, so the original error propagates. Tests (red first): tests/test_subscriber_migration_rollback.py - a failed BEGIN surfaces 'database is locked', not 'cannot rollback' (reproduces the production error before the fix); - an error inside the transaction still rolls back and leaves the DB unchanged; - a needed legacy rename is still committed. Subscriber suites: 61 passed. The 2 test_deploy_environment_boots... tests fail identically on unpatched main (they need the deployment environment). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec
echo-fleet-builder Bot
pushed a commit
that referenced
this pull request
Nov 3, 2026
…ompensation Port PR #40 and PR #41 onto deployed d8ad9fd without unrelated main changes. Retain historical crash credits by tenant/run identity and use a separate run-compensation namespace for new credits. Add atomicity, concurrency, legacy-credit, original-period, collision-order, and migration rollback tests. Verified: 705 passed, 1 Docker-dependent skip; P1/P6/P7 acceptance green; 87 main integration tests passed; 7/7 isolated mutation checks killed. No deployment, production state repair, or live compensation authorized.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes Echo fleet item #46912: the
echo-certforge-dispatcher.servicerestart storm on 2026-10-19 (incident #49782, auto-resolved).Root cause. In
SubscriberGovernance._migrate_legacy_platform_tables, theexcept BaseExceptionhandler always ranROLLBACK. WhenBEGIN IMMEDIATEitself fails, there is no transaction open, soROLLBACKraisessqlite3.OperationalError: cannot rollback - no transaction is active. A failedBEGIN IMMEDIATEhappens, for example, when another process holds the write lock past the 10 s busy timeout and SQLite reportsdatabase is locked. That second error replaced the real one and crashed the dispatcher at start-up. This is the traceback in the incident.Fix. Roll back only when
connection.in_transactionis set, so the original error propagates and is the one logged.Tests
New
tests/test_subscriber_migration_rollback.py, written to fail first:BEGIN IMMEDIATEsurfacesdatabase is locked, notcannot rollback. Before the fix this test reproduces the production error exactly.With
tests/test_p7_subscriber_governance.pyas well: 61 passed. The 2test_deploy_environment_boots_and_authenticated_smoke_is_green[...]cases fail identically on unpatchedmain(58 passed + the same 2), because they need the deployment environment.Deployment
Production currently runs release
d8ad9fd(2026-09-26), which has the same code, so the defect is latent there. It only triggers when start-up loses the write-lock race. I did not hot-patch the certification authority. This change should ship throughdeploy/deploy_forge.shonce reviewed.🤖 Generated with Claude Code
https://claude.ai/code/session_01C4YSVhYLwZidNF5snQ4eec