Expired activity lease repair bypasses schedule-to-close timeout

[rmcdaniel]

Observed defect

With published Workflow 2.0.10, a worker crash followed by a repair pass after both the activity lease and schedule-to-close deadline have expired can re-run the activity and complete the workflow instead of recording a terminal timeout.

Reproduced using separate PHP processes, a persisted SQLite database and a real SIGKILL after a generated external-effect marker was written. No live customer runtime or data was involved.

Reproduction

1. Start a workflow with an activity configured with maxAttempts=3, startToCloseTimeout=55 and scheduleToCloseTimeout=600.
2. Claim/run the activity, write an idempotent external-effect marker, and kill the PHP worker before recording completion.
3. Start a fresh process, advance the test clock beyond the lease and the 600-second overall deadline, and run the normal workflow:v2:repair-pass.
4. Process the repaired database queue.

Observed repair report: one repaired task; one activity-timeout candidate; zero timeouts enforced; reason current_attempt_not_running.
Observed history: a second ActivityStarted, ActivityCompleted, then WorkflowCompleted. No ActivityTimedOut is recorded.

TaskRepair closes the expired attempt before the timeout scan. ActivityTimeoutEnforcer then rejects that expired current attempt, and ActivityTaskClaimer can claim the repaired task without enforcing the existing overall deadline.

This is distinct from #498: the exception class now exists, but this path never creates the timeout event.

Acceptance

• An elapsed schedule-to-close deadline remains authoritative through lease repair and fresh-process recovery.
• No activity handler or later workflow effects run after the deadline in this reproduction.
• Record exactly one terminal timeout and restore it normally during workflow replay.
• Preserve retry behavior before the overall deadline, attempt fencing, lock order and stale-completion rejection.
• Cover the ordinary repair pass and the shared claim/enforcement paths used by embedded workers and Server.
• Publish the fix and verify its affected consumers with published artifacts.

[rmcdaniel]

The fix is merged in #504 (9112315). All 28 MySQL activity-timeout tests passed, including the new expired-lease cases and concurrent heartbeat/enforcement lock-order test. The three new regressions fail on the previous release. Both real worker-termination/fresh-process consumer cases also pass with the candidate. Full main-branch validation is running before publishing 2.0.11; Server and application consumers will be updated and checked against the published package. Keeping this issue open until delivery is verified.

[rmcdaniel]

Fixed and delivered in Workflow 2.0.11 through #504.

Verification includes the expired-lease/deadline regression tests, MySQL concurrency coverage, and real SIGKILL followed by a fresh PHP/database reload: recovery before the deadline reuses the existing effect; recovery after it records the terminal timeout without another handler execution. The published Laravel upgrade matrix passed all 16 journeys.

Downstream delivery is verified:

• Server 2.3.8 includes Workflow 2.0.11. Its published image passed all activity checks and 31 PHP/Python/Rust replay checks.
• Sample App uses 2.0.11; main application/polyglot checks and AMD64/ARM64 prepared-image publication, anonymous qualification and channel promotion passed.

SDK/CLI protocol and implementation did not change, so they do not need version-only releases. Waterline uses Workflow as a development dependency rather than bundling a replacement runtime. Managed runtime deployments retain their separate qualification/rollout record.

The separately identified SQLite completion-upload CI expectation is tracked in durable-workflow/server#154; it does not reopen this repaired activity-deadline defect.