fix(rsqueue): roll back a partially completed permit sweep - #299
Merged
Conversation
DatabaseQueueSweeperTask.Run deletes every expired permit inside one transaction, and the deferred CompleteTransaction reads the function's `err` variable to decide between COMMIT and ROLLBACK. The delete inside the loop used `:=`, which redeclared `err` rather than assigning to it, so a failure part-way through returned with the outer error still nil and the transaction committed. The result was a partially swept set of permits, made permanent and reported to the caller as a clean sweep: the permits deleted before the failure were released while the remainder were left held, with nothing recording that the sweep had not finished. Assign to the outer `err` so a failed delete rolls the whole sweep back. The sweep is idempotent and runs on a schedule, so the next pass retries the same work. TestSweepDeleteErrRollsBack asserts on the error handed to CompleteTransaction rather than on a delete count, because the buggy and fixed versions attempt exactly the same deletes and both return at the first failure -- the only observable difference is whether the partial result is kept. TestSweepOkCommits pins the successful case alongside it, so the new assertion cannot pass for the wrong reason. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
DatabaseQueueSweeperTask.Runcould commit a partially completed permit sweep and report it to the caller as a clean one.Rundeletes every expired permit inside a single transaction, and the deferredCompleteTransaction(&err)reads the function'serrvariable to decide between COMMIT and ROLLBACK. The delete inside the loop was written with:=:which declares a new
errscoped to theifbody instead of assigning to the one the deferred call watches. So a failure part-way through the loop returned with the outererrstillnil, and the transaction committed.The observable result: the permits deleted before the failure were released and made permanent, the remainder were left held, and nothing recorded that the sweep had not finished.
Runhas no error return, so the only signal was aslog.Debugline.Fix
Assign to the outer
err, so a failed delete rolls the whole sweep back. The sweep is idempotent and scheduled, so the next pass retries the same work — a rollback costs one interval, where the previous behaviour left the queue in a state no caller asked for.Tests
TestSweepDeleteErrRollsBackasserts on the error handed toCompleteTransactionrather than on a delete count. That distinction is the whole point: the buggy and fixed versions attempt exactly the same deletes and both return at the first failure, so no count can tell them apart. The only observable difference is whether the partial result is kept.To make that assertable,
QueueTestStore.CompleteTransactionnow records the error it was given; it was previously an empty method, so no test could see commit-versus-rollback at all. The field is additive and no existing test reads it.TestSweepOkCommitspins the successful case beside it, so the new assertion cannot pass for the wrong reason — without it, a fixture that never swept anything would satisfy a rollback assertion trivially.Verified by reverting only
sweeper.goand re-running:TestSweepDeleteErrRollsBackfails withobtained = nilagainst the expected delete error, and the other three cases still pass.Checks
go test ./pkg/rsqueue/...— passes.go build ./...andgo vet ./...— clean../scripts/fmt-check.sh,./scripts/header-check.sh,./scripts/test-wiring.sh— pass.go test ./...—pkg/rsnotify/listeners/postgrespgxtimes out locally because it needs a reachablepostgreshost; that is thejust test-integrationpath and is untouched by this change.Notes
I checked for the same shadowing elsewhere in the module. The only other
CompleteTransaction(&err)user isexamples/cmd/markdownRenderer/store/queue.go, which assigns rather than redeclares, and itsQueuePopuses gorm's manageddb.Transaction(func(tx) error), which is correct by construction. This was the only instance.Worth considering separately:
govet'sshadowcheck is not enabled in.golangci.yml, and it catches this class directly. Enabling it would likely need a pass over existing findings, so it is not bundled here.🤖 Generated with Claude Code