fix(workflow): keep the workflow worker working across sidecar reconnects by famarting · Pull Request #1312 · dapr/python-sdk

famarting · 2026-10-19T10:22:31Z

Description

The workflow worker reconnects by itself when the Dapr sidecar ends its work-item stream (a sidecar restart, a proxy or load balancer recycling the connection, a network change). Four bugs around that reconnect can make the worker stop receiving work, lose results, or crash the process. One commit per fix:

1. The reconnect Hello has no deadline, so a reconnect can hang forever.
Every new connection is checked with stub.Hello(Empty()) without a timeout (the keepalive's Hello already uses timeout=10). After a host sleep or a network change, the new connection can be left half-open and that Hello never returns. It runs on the worker's event loop, so the worker never reconnects, and the sidecar has no worker for the app while the process looks healthy.
Reproduce: run a worker against a sidecar, drop the packets between them (for example with a firewall rule, or suspend the host) so the stream ends without a clean close. The worker logs that the stream ended and never logs "Created fresh connection" again.
Fix: the Hello gets a 30s deadline (longer than the keepalive's because it also covers the TCP and TLS handshakes), so a stuck attempt fails and the existing retry with backoff runs. A failed attempt's channel is now closed instead of leaked.

2. is_worker_ready() stays True after the stream is lost.
The ready flag is set on the first connect and never cleared, so callers cannot tell a worker that is receiving work from one that has been cut off from the sidecar.
Reproduce: start a WorkflowRuntime, stop the sidecar; is_worker_ready() keeps returning True.
Fix: the flag is cleared when the stream is dropped and set again when the next stream opens. This matches what WorkflowRuntime.wait_for_worker_ready() documents ("the worker's gRPC stream is ready to receive work items"); start() still waits for the first stream exactly as before.

3. A result that outlives a reconnect is lost.
Each work item carries the stub of the connection it arrived on and sends its result on it. After a reconnect the old channel is closed 10 seconds later, so any activity or workflow step that runs longer than that sends its result on a closed channel. The ValueError is logged at debug level and the result is dropped, so the activity runs again from the start.
Reproduce: register an activity that sleeps 60s, start a workflow that calls it, and restart the sidecar (or kill the gRPC connection) while it runs. The activity finishes, but its result never reaches the sidecar.
Fix: results (and the history fetch on a cache miss) are sent on the worker's current connection. If the result is ready while the worker is between connections, it waits up to 30s for the new one; if the worker is stopping it gives up and the sidecar re-dispatches the work item.

4. A call on a closed channel can crash the process with SIGSEGV.
grpcio raises ValueError for a call on a closed channel, but not when the channel has client interceptors (the workflow runtime always adds one). An intercepted call first registers its method on the underlying channel (Channel._get_registered_call_handle), which locks a mutex inside the already freed native channel, and the process dies with a segmentation fault. Before fix 3 this happened every time a result was sent after its channel was closed; independently, stop() closed the current channel while a result send, a keepalive Hello, the connect Hello or GetWorkItems could be starting on it.
Reproduce: same as 3, with the old channel's close happening before the activity finishes. The process exits with code 139. (Verified locally against a real in-process gRPC server: exit 139 on main, result delivered with this PR.)
Fix: every call the worker makes goes through one helper that only hands out the current connection and counts the calls in flight on each channel. A channel is closed only once it is out of use and its count is zero. stop() waits up to 5 seconds for in-flight calls and then closes anyway; by then any such call has long started, so the close only cancels it.

Tests for each fix are in tests/ext/workflow/durabletask/test_worker_reconnect.py, using a fake sidecar in the style of test_worker_health_ping.py. Two existing tests were adapted because the internal handlers no longer take the work item's stub.

Issue reference

Please reference the issue this PR will close: no issue filed yet; opened as a draft to discuss the fixes first.

Checklist

Code compiles correctly
Created/updated tests
Extended the documentation

🤖 Generated with Claude Code

The worker checks each new connection to the sidecar with a Hello call that had no deadline. After a host sleep or a network change the new connection can be left half-open, and that Hello then waits forever. It runs on the worker's event loop, so the worker never reconnects while the process looks healthy. Give the Hello a 30 second deadline, so a stuck attempt fails and the existing retry loop tries again. Also close the channel of a failed attempt instead of dropping it, so repeated failures do not leak channels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com>

is_worker_ready() was set on the first connect and never cleared, so it kept returning True while the worker had lost its work-item stream and was trying to reconnect. Nothing outside the worker could tell a worker that receives work from one that has been cut off from the sidecar. Clear the ready flag whenever the stream is dropped; it is set again when the next stream opens. This matches what WorkflowRuntime documents for wait_for_worker_ready(): that the stream is ready to receive work items. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com>

Each activity and workflow work item carried the stub of the connection it arrived on, and sent its result through that stub. When the sidecar ends the work-item stream the worker opens a new connection and closes the old channel ten seconds later. Any work item that runs longer than that sends its result on a closed channel: grpc raises ValueError, the worker logs it at debug level, and the result is lost. The sidecar then runs the activity again from the start. Send results (and the history fetch for a cache miss) through the worker's current connection instead. If the result is ready while the worker is between connections, wait up to 30 seconds for the next one; if the worker is stopping, give up at once and let the sidecar re-dispatch the work item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com>

…l on it grpcio does not refuse a call on a closed channel when the channel has client interceptors (the workflow runtime always adds one). The intercepted call registers its method on the underlying channel first, which locks a mutex inside the already freed native channel, and the whole process dies with SIGSEGV instead of getting a ValueError. The worker closed channels without regard to calls about to start on them: the old channel ten seconds after a reconnect, and the current one at once in stop(), racing result sends, the keepalive Hello, the connect Hello and GetWorkItems. Count the calls in flight on each channel. Every call the worker makes now takes the channel through _sidecar_stub, which only hands out the current connection, and a channel is closed only after it is retired and its count drops to zero. stop() waits up to five seconds for those calls and then closes the channel anyway; by then any such call has long started, so closing it only cancels it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com>

codecov · 2026-10-19T10:24:51Z

Codecov Report

❌ Patch coverage is 91.07143% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.27%. Comparing base (d0318b9) to head (b236f85).

Files with missing lines	Patch %	Lines
dapr/ext/workflow/_durabletask/worker.py	91.07%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1312      +/-   ##
==========================================
+ Coverage   86.98%   87.27%   +0.28%     
==========================================
  Files         163      163              
  Lines       12997    13069      +72     
==========================================
+ Hits        11306    11406     +100     
+ Misses       1691     1663      -28

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

🟡 Changes recommended

Connection publication and cleanup still permit lost results, blocked shutdown, and unsafe calls after channel closure.

3 open issues

Expose stubs only after reconnect Hello succeeds · New
Synchronize fallback RPCs with channel closure · New
Shutdown can hang indefinitely on retired RPCs · New

What changed in this PR

This PR improves the Dapr workflow worker’s recovery from sidecar disconnects.

Changes:

Adds connection-check deadlines and resets readiness during reconnects.
Routes results through the current connection and tracks channel usage.
Adds reconnect regression tests and updates existing tests and guidance.

File	Description
tests/ext/workflow/durabletask/test_worker_reconnect.py	Adds reconnect, result-delivery, and channel-cleanup tests.
tests/ext/workflow/durabletask/test_worker_history_cache.py	Adapts history tests to worker-managed connections.
tests/ext/workflow/durabletask/test_async_dispatch_regression.py	Updates async dispatch tests for revised handlers.
dapr/ext/workflow/AGENTS.md	Documents current-connection result delivery.
dapr/ext/workflow/_durabletask/worker.py	Implements reconnect and channel-lifetime changes.

_{Review effort: Balanced}

JoshVanL · 2026-10-19T11:56:05Z

            finally:
                try:
-                    old_channel.close()
+                    self._close_channel(old_channel, drain_timeout=None)


The deferred close waits forever for a result RPC that never returns, which keeps the old channel open and blocks stop().

_close_channel(old_channel, drain_timeout=None) waits until _calls_in_flight[old_channel] reaches zero. CompleteActivityTask, CompleteOrchestratorTask and GetInstanceHistory are sent with no deadline, so a call to a sidecar that accepts the request and never answers (half-open connection, wedged sidecar) holds the count at one indefinitely. The channel is never closed, the pool thread running _deliver_activity_response stays blocked in the RPC, and thread_pool.shutdown(wait=True) at line 2884 never returns, so stop() hangs. On main the channel was closed unconditionally after the 10s grace, which cancelled the stuck call and released the thread.

Repro: publish a stub whose CompleteActivityTask blocks until the channel is closed, submit _deliver_activity_response to the pool, then _retire_connection(), _schedule_deferred_channel_close(..., grace_timeout=0.1), _set_connection(new, ...) and stop(). On this branch stop() is still blocked after 15s with the old channel open. On main it returns.

Suggested fix: keep the close bounded. _close_channel already returns as soon as the count hits zero, so the drain timeout can replace the _shutdown.wait as the grace period:

def _deferred_close(): try: self._close_channel(old_channel, drain_timeout=grace_timeout) self._logger.debug('Deferred channel close completed') except Exception as e: self._logger.debug(f'Error during deferred channel close: {e}')

A deadline on the result RPCs (separate comment) also bounds this, but the close should not depend on it.

I added the deadlines (e68ae57), so a wedged sidecar now releases the call after _RESULT_SEND_TIMEOUT and the close and stop() finish. I did not make the close itself time out, and I think it should not. Closing while the count is above zero is the SIGSEGV this PR fixes: a holder can make a second call under the same acquisition (the oversized-result fallback), or be between acquiring and starting its call, and a close that lands there crashes the process (Copilot raised the same thing on line 1047). So the deadline is the bound, and _close_channel documents that it relies on it. A fake whose CompleteActivityTask ignores timeout will still block, but a real grpcio call cannot. Leaving this open in case you see a case the deadline does not cover.

JoshVanL · 2026-10-19T11:56:06Z

+                del self._calls_in_flight[channel]
+            self._connection_lock.notify_all()
+
+    def _close_channel(self, channel: grpc.Channel, drain_timeout: Optional[float]) -> None:


The in-flight count, the retire and publish handshake, and the deferred close threads exist only because the worker replaces its gRPC channel on every stream end.

A grpc.Channel reconnects on its own. What has to be re-issued after a stream ends is the GetWorkItems call. With one channel for the worker's lifetime, results always go on the current connection by construction, stop() closes exactly one channel, the SIGSEGV case in this docstring cannot occur, and the edge cases in the other comments go away with the code that has them.

If recreating the channel is needed for a specific sidecar or proxy behaviour, a one-line comment next to retire_connection saying which would stop the next person from trying the simpler design blind. Not blocking.

Agreed that a single channel would remove most of this. Recreating the channel on every stream end predates this PR: it came in with the vendored durabletask-python worker (#963), and I do not know which sidecar or proxy behaviour it was added for, so I would rather not guess in a comment. Switching to one channel changes how the worker reconnects for every user, so I think it belongs in its own PR where it can be tested against real reconnects (sidecar restart, L7 proxy idle timeout) rather than folded into this fix. Leaving this open for that decision.

…very call on it Addresses review feedback on the reconnect fix: - Hello runs on the new connection before it is published, so a result waiting for a reconnect is never sent on a connection that then fails. - Result sends and the history fetch get a 30s deadline. A retired channel is closed only once no call is in flight on it, so these deadlines are what bound that close and stop(); the close itself never runs early, since that could land between a result and its oversized-result fallback. - stop() keeps the connection open for its drain, so a result that becomes ready while the worker stops is still delivered. - Releasing a call wakes waiters only when the channel's count reaches zero. - The run loop tracks a connected flag instead of an unused stub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com>

…ects (#1312) (#1324) * fix(workflow): put a deadline on the worker's reconnect Hello The worker checks each new connection to the sidecar with a Hello call that had no deadline. After a host sleep or a network change the new connection can be left half-open, and that Hello then waits forever. It runs on the worker's event loop, so the worker never reconnects while the process looks healthy. Give the Hello a 30 second deadline, so a stuck attempt fails and the existing retry loop tries again. Also close the channel of a failed attempt instead of dropping it, so repeated failures do not leak channels. * fix(workflow): report a worker as not ready while its stream is down is_worker_ready() was set on the first connect and never cleared, so it kept returning True while the worker had lost its work-item stream and was trying to reconnect. Nothing outside the worker could tell a worker that receives work from one that has been cut off from the sidecar. Clear the ready flag whenever the stream is dropped; it is set again when the next stream opens. This matches what WorkflowRuntime documents for wait_for_worker_ready(): that the stream is ready to receive work items. * fix(workflow): send work-item results on the worker's current connection Each activity and workflow work item carried the stub of the connection it arrived on, and sent its result through that stub. When the sidecar ends the work-item stream the worker opens a new connection and closes the old channel ten seconds later. Any work item that runs longer than that sends its result on a closed channel: grpc raises ValueError, the worker logs it at debug level, and the result is lost. The sidecar then runs the activity again from the start. Send results (and the history fetch for a cache miss) through the worker's current connection instead. If the result is ready while the worker is between connections, wait up to 30 seconds for the next one; if the worker is stopping, give up at once and let the sidecar re-dispatch the work item. * fix(workflow): never close a channel while the worker may start a call on it grpcio does not refuse a call on a closed channel when the channel has client interceptors (the workflow runtime always adds one). The intercepted call registers its method on the underlying channel first, which locks a mutex inside the already freed native channel, and the whole process dies with SIGSEGV instead of getting a ValueError. The worker closed channels without regard to calls about to start on them: the old channel ten seconds after a reconnect, and the current one at once in stop(), racing result sends, the keepalive Hello, the connect Hello and GetWorkItems. Count the calls in flight on each channel. Every call the worker makes now takes the channel through _sidecar_stub, which only hands out the current connection, and a channel is closed only after it is retired and its count drops to zero. stop() waits up to five seconds for those calls and then closes the channel anyway; by then any such call has long started, so closing it only cancels it. * fix(workflow): verify a connection before results use it, and bound every call on it Addresses review feedback on the reconnect fix: - Hello runs on the new connection before it is published, so a result waiting for a reconnect is never sent on a connection that then fails. - Result sends and the history fetch get a 30s deadline. A retired channel is closed only once no call is in flight on it, so these deadlines are what bound that close and stop(); the close itself never runs early, since that could land between a result and its oversized-result fallback. - stop() keeps the connection open for its drain, so a result that becomes ready while the worker stops is still delivered. - Releasing a call wakes waiters only when the channel's count reaches zero. - The run loop tracks a connected flag instead of an unused stub. --------- (cherry picked from commit 9136b37) Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> Signed-off-by: dapr-bot <dapr-bot@users.noreply.github.com> Co-authored-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

famarting and others added 4 commits October 19, 2026 11:14

JoshVanL requested a balanced review from Copilot October 19, 2026 10:27

Copilot started reviewing on behalf of JoshVanL October 19, 2026 10:28

View session

Copilot AI reviewed Oct 19, 2026

View reviewed changes

Comment thread dapr/ext/workflow/_durabletask/worker.py Outdated

Comment thread dapr/ext/workflow/_durabletask/worker.py Outdated

Comment thread dapr/ext/workflow/_durabletask/worker.py Outdated

JoshVanL requested changes Oct 19, 2026

View reviewed changes

famarting marked this pull request as ready for review October 19, 2026 13:43

famarting requested review from a team as code owners October 19, 2026 13:43

This was referenced Oct 19, 2026

refactor(workflow): keep one gRPC channel for the worker's lifetime #1317

Open

test(actor): retry the app-channel listener test on a taken port #1318

Merged

CasperGN self-assigned this Oct 19, 2026

Merge branch 'main' into fix/workflow-worker-reconnect

b236f85

famarting requested a review from JoshVanL October 20, 2026 07:18

JoshVanL approved these changes Oct 20, 2026

View reviewed changes

CasperGN approved these changes Oct 20, 2026

View reviewed changes

CasperGN added this pull request to the merge queue Oct 20, 2026

Merged via the queue into dapr:main with commit 9136b37 Oct 20, 2026
22 checks passed

famarting deleted the fix/workflow-worker-reconnect branch October 20, 2026 15:17

sicoyle added this to the v1.19 milestone Oct 20, 2026

CasperGN removed their assignment Oct 20, 2026

nelson-parente added the backport release-1.19 label Oct 21, 2026

dapr-bot mentioned this pull request Oct 21, 2026

[Backport release-1.19] fix(workflow): keep the workflow worker working across sidecar reconnects #1324

Merged

Navigation Menu

Repository navigation

fix(workflow): keep the workflow worker working across sidecar reconnects - #1312

fix(workflow): keep the workflow worker working across sidecar reconnects#1312
CasperGN merged 6 commits into
dapr:mainfrom
famarting:fix/workflow-worker-reconnect

famarting commented Oct 19, 2026

Uh oh!

codecov Bot commented Oct 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Oct 19, 2026

Uh oh!

famarting Oct 19, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Oct 19, 2026

Uh oh!

famarting Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

famarting commented Oct 19, 2026

Description

Issue reference

Checklist

Uh oh!

codecov Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

famarting Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JoshVanL Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

famarting Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Oct 19, 2026 •

edited

Loading