Skip to content

Preserve activity deadlines through lease repair - #504

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/activity-timeout-lease-repair
Sep 9, 2026
Merged

Preserve activity deadlines through lease repair#504
rmcdaniel merged 3 commits into
mainfrom
fix/activity-timeout-lease-repair

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Problem

Addresses #503. Keep that issue open until the patch is published and affected consumers are verified.

An expired activity lease can be repaired before the timeout sweep. The expired attempt was rejected by timeout enforcement, while task claiming could replace its elapsed deadlines.

Change

  • Preserve the existing identity and attempt-number fencing checks while allowing the matching expired attempt to reach timeout enforcement.
  • Reuse the same deadline predicate at claim time so pending/repaired work cannot overwrite an expired deadline.
  • Leave timeout retry/failure recording with the existing enforcer. No new scheduler, lock order or protocol change.

Validation

The three new reproduction cases failed on unchanged main. With the fix, all seven focused lease/timeout-replay cases pass (56 assertions): ordinary repair, state already repaired by an older worker, overdue initial claim, late stale completion, and ordinary/local timeout restoration. ECS and diff checks pass. Broader timeout tests, static analysis and repository CI are in progress.

Published-artifact follow-through: Workflow patch, downstream Server rebuild/release and consuming application qualification; unchanged SDKs do not need synchronized releases. Exact published PHP/Python/Rust activity/replay checks must be recorded before delivery is claimed.

@rmcdaniel
rmcdaniel marked this pull request as ready for review September 9, 2026 18:50
@rmcdaniel

Copy link
Copy Markdown
Member Author

Local validation: changed-file PHPStan and ECS pass. The timeout/lease/replay tests pass with 32 tests / 188 assertions; one concurrent database lock-order case is skipped under SQLite and remains part of the MySQL qualification, not counted as a pass here. All three new cases failed on unchanged main. Repository PR checks are running, including the real MySQL feature shard. The change does not alter wire encoding, timeout identity or row-lock acquisition order.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Fresh-process consuming regression also passes with this candidate source: a real SIGKILL after an idempotent external-effect marker, followed by a new PHP process and normal database-queue repair. Recovery before the overall deadline completes with the original operation identity. Recovery after the overall deadline records and restores the terminal timeout without running another activity effect. This was candidate validation, not published-artifact evidence. Generated databases, container and volume were removed.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Checked the actual PR job logs: the first bounded MySQL pass ran the four fallback cases, not the new database-backed unit file. The regressions now live alongside V2ActivityTimeoutTest, so the existing changed-feature selection runs all timeout cases, including MySQL concurrency, without adding test-selector machinery. Local candidate cold-process results remain valid; the new-head CI result is still pending.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Confirmed the changed-feature MySQL job actually executed all28 activity-timeout cases (181 assertions), including both expired-lease regressions, overdue initial claim and concurrent heartbeat/enforcement lock order; no skips/failures. Preparing patch2.0.11 in this same PR so merge, main validation and immutable publication follow one reviewed change. Private consuming-application rollout remains gated on the published package, not the source override used in the crash proof.

@rmcdaniel

Copy link
Copy Markdown
Member Author

Release-head code/contract checks are passing. The dependency-security job stopped because Packagist returned no response within its10-second advisory-request timeout for Laravel13, after the earlier majors were audited; no new advisory was reported. Retrying that failed job once without changing or disabling the audit.

@rmcdaniel
rmcdaniel merged commit 9112315 into main Sep 9, 2026
16 of 17 checks passed
@rmcdaniel
rmcdaniel deleted the fix/activity-timeout-lease-repair branch September 9, 2026 19:04
@rmcdaniel

Copy link
Copy Markdown
Member Author

Workflow 2.0.11 is published at 9112315, verified on Packagist, and all 16 published Laravel upgrade journeys passed: https://github.com/durable-workflow/workflow/actions/runs/34394061181. The actual consumer cold-replay checks now pass using the installed published package, without source overrides. Server image follow-through is in durable-workflow/server#153. SDK/CLI packages do not need a release because their code and portable protocol are unchanged. Sample App still needs its pinned embedded dependency updated; application deployments are tracked in their owning records. Issue #503 stays open until the remaining downstream delivery is verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants