Skip to content

Preserve activity completion authority and concurrent upload budgets - #148

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/completion-upload-task-authority
Sep 8, 2026
Merged

Preserve activity completion authority and concurrent upload budgets#148
rmcdaniel merged 3 commits into
mainfrom
fix/completion-upload-task-authority

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 8, 2026

Copy link
Copy Markdown
Member

Problem

Refs #146. Published-artifact testing exposed two independent failures while draining storage:

  1. A heartbeating activity retains its task lease while its synchronous worker's polling-roster entry ages. The upload path incorrectly rejected that valid completion.
  2. Concurrent first reservations for different activity budgets could deadlock in MySQL because locking reads of absent IDs acquired overlapping InnoDB gap locks.

Change

  • Activity uploads use the current namespace/task/attempt/owner and unexpired activity lease, matching ordinary activity completion. Workflow/query roster requirements remain unchanged.
  • Use Eloquent createOrFirst before the locking read for a new budget. Existing reservations are read with a locking current read; immutable slots, byte/object limits, object locking and transaction retries are preserved.
  • Prepare Server 2.3.6 and Helm 0.1.85 through the existing source-release manifest generator.

No SDK change, lease renewal, expanded budget, new retry engine or storage-fence bypass.

Verification

  • Activity/workflow/query roster-state regressions and existing invalid-lease, storage admission, cleanup and idempotency cases. The final focused run passed 100 tests / 3,695 assertions; activity idempotency also passed in the preceding 101-test run.
  • Eight fresh-process database cases passed with 236 assertions: SQLite, MySQL 8.0.46 and PostgreSQL 16.4, including eight independent concurrent budgets on each native database and fresh-process retries.
  • Source-patched native storage-pressure run: 24 original activity results acknowledged, then all workflows completed after admission recovered. Fresh SDK reads after restarting Server, workers, scheduler, MySQL and Redis preserved exact bytes, original attempt IDs and one recorded side effect per activity. This is source qualification, not published-image proof or a capacity claim.
  • Pint, source-release synchronization and normal repository CI at 9a660239086a95f0f21f4ecfc5074dcc1131d5cc passed.
  • Repeat with the published image and fresh external objects before closing Storage draining rejects payload uploads needed to finish already-leased work #146.

The earlier published failure and intermediate 23/24 failure remain recorded failures. No production deployment is claimed.

@rmcdaniel rmcdaniel changed the title Use activity task authority for completion uploads Preserve activity completion authority and concurrent upload budgets Sep 8, 2026
@rmcdaniel
rmcdaniel marked this pull request as ready for review September 8, 2026 18:50
@rmcdaniel
rmcdaniel merged commit fbb7c6e into main Sep 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants