Skip to content

Refuse to retry a job a worker is still running - #42

Merged
alexeybe1kin merged 2 commits into
mainfrom
glm/memorygate-retry-live-lease
Nov 1, 2026
Merged

alexeybe1kin merged 2 commits into
mainfrom
glm/memorygate-retry-live-lease

Conversation

@alexeybe1kin

Copy link
Copy Markdown
Collaborator

POST /jobs/{id}/retry reset any job to pending and cleared its lease, even one a worker held a live lease on, so a second worker could claim and process the same evidence while the first was still running (L-04 family).

  • requeue(db, job_id) updates in one statement, only when the job is not processing, has no lease, or the lease has expired; the route answers 409 otherwise.
  • Tests: live lease refused and left untouched; failed job and expired lease requeued; missing job is false.
  • Mutation proof: dropping the lease condition from the WHERE makes test_retry_refuses_a_job_with_a_live_lease fail.
  • pytest tests/test_processing_worker.py + retry tests: 12 passed; ruff clean.

Written by GLM (worker-mcp), reviewed by Claude.

🤖 Generated with Claude Code

alexeybe1kin and others added 2 commits November 1, 2026 17:16
- Add requeue helper that only clears lease when job is not processing,
  has no lease, or lease is expired
- Update retry_job route to use requeue and return 409 when live lease exists
- Add tests for live lease refusal, failed job requeue, expired lease requeue,
  and missing job handling
- Prevents double-processing of evidence when retry is called during active job

Fixes race condition where retry could queue same job for multiple workers
simultaneously, violating lease protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alexeybe1kin
alexeybe1kin merged commit 80861d8 into main Nov 1, 2026
3 checks passed
@alexeybe1kin
alexeybe1kin deleted the glm/memorygate-retry-live-lease branch November 1, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant