Skip to content

Preserve worker requests through storage admission pauses - #57

Merged
rmcdaniel merged 1 commit into
mainfrom
fix/storage-admission-recovery
Sep 7, 2026
Merged

Preserve worker requests through storage admission pauses#57
rmcdaniel merged 1 commit into
mainfrom
fix/storage-admission-recovery

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Sep 7, 2026

Copy link
Copy Markdown
Member

Problem

Part of durable-workflow/server#131. Bounded HTTP retries exhaust during a storage pause; poll loops can rotate claim identities, and completed activity results can be dropped. Retrying a whole completion method also repeats external-payload uploads.

Changes

  • Extend the existing transport retry policy only inside Worker execution, including the separate query thread. Direct clients and control-plane calls retain bounded retries.
  • Validate the explicit storage-refusal contract; preserve the prepared request body, poll/task/attempt identity, and uploaded payload reference.
  • Interrupt waits on shutdown/cancellation; preserve authentication and expired-lease rejection. Apply the run_until deadline across registration and admission pauses.
  • Do not convert storage rejection into an application failure or a successful completion. The standalone legacy validation poll has no replay identity and remains fail-closed; workers use the identity-bearing multiplexed poll.

Validation

  • 1,370 non-integration tests passed locally; Ruff, mypy and regression-corpus validation passed.
  • 80 focused storage/cold-replay cases passed. Replacing only the transport retry implementation with the base version makes the cold-completion check fail with four replays instead of one.
  • Public Python 3.10-3.12, Server integration, package, portal and required checks passed.
  • Execution counts, byte-identical acknowledgements, uploaded references, separate query-thread context, registration/poll/completion timeouts, explicit stop, authentication and expired leases are covered.

This PR does not enable Server admission, publish a package, or claim native storage-limit qualification. Rust recovery and real capacity-limited writer tests remain on server#131.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Local validation at aebf235:

  • Full non-integration suite: 1,370 passed; 15 live-runtime integration cases deselected.
  • Ruff and strict mypy: clean. Regression-corpus policy: pass, with a new cold-completion fixture.
  • Negative control: using the transport retry implementation from origin/main with the new cold-completion consumer fails because the workflow is replayed four times instead of once. The candidate passes with one replay and eight byte-identical completion submissions. This isolates the transport defect without changing fixture expectations.
  • Real Worker execution covers activity success/failure and heartbeat pressure; query polling runs in its separate thread; validation approval/rejection invokes the validator once. Runtime-owned external payload upload is accepted once, and completion retries reuse its reference.
  • Registration, poll, and completion pressure respect run_until timeout; explicit stop interrupts registration; authentication and expired-lease errors remain terminal.

Public Python 3.10-3.12, packaging and portal checks are running. These are HTTP fault-injection tests, not the outstanding native database-full qualification on server#131.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 7, 2026 23:03
@rmcdaniel
rmcdaniel merged commit 0def8d8 into main Sep 7, 2026
13 checks passed
@rmcdaniel
rmcdaniel deleted the fix/storage-admission-recovery branch September 7, 2026 23:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants