Repository navigation
[Backport release-1.19] fix(workflow): keep the workflow worker working across sidecar reconnects - #1324
Merged
Conversation
…ects (#1312) * fix(workflow): put a deadline on the worker's reconnect Hello The worker checks each new connection to the sidecar with a Hello call that had no deadline. After a host sleep or a network change the new connection can be left half-open, and that Hello then waits forever. It runs on the worker's event loop, so the worker never reconnects while the process looks healthy. Give the Hello a 30 second deadline, so a stuck attempt fails and the existing retry loop tries again. Also close the channel of a failed attempt instead of dropping it, so repeated failures do not leak channels. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> * fix(workflow): report a worker as not ready while its stream is down is_worker_ready() was set on the first connect and never cleared, so it kept returning True while the worker had lost its work-item stream and was trying to reconnect. Nothing outside the worker could tell a worker that receives work from one that has been cut off from the sidecar. Clear the ready flag whenever the stream is dropped; it is set again when the next stream opens. This matches what WorkflowRuntime documents for wait_for_worker_ready(): that the stream is ready to receive work items. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> * fix(workflow): send work-item results on the worker's current connection Each activity and workflow work item carried the stub of the connection it arrived on, and sent its result through that stub. When the sidecar ends the work-item stream the worker opens a new connection and closes the old channel ten seconds later. Any work item that runs longer than that sends its result on a closed channel: grpc raises ValueError, the worker logs it at debug level, and the result is lost. The sidecar then runs the activity again from the start. Send results (and the history fetch for a cache miss) through the worker's current connection instead. If the result is ready while the worker is between connections, wait up to 30 seconds for the next one; if the worker is stopping, give up at once and let the sidecar re-dispatch the work item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> * fix(workflow): never close a channel while the worker may start a call on it grpcio does not refuse a call on a closed channel when the channel has client interceptors (the workflow runtime always adds one). The intercepted call registers its method on the underlying channel first, which locks a mutex inside the already freed native channel, and the whole process dies with SIGSEGV instead of getting a ValueError. The worker closed channels without regard to calls about to start on them: the old channel ten seconds after a reconnect, and the current one at once in stop(), racing result sends, the keepalive Hello, the connect Hello and GetWorkItems. Count the calls in flight on each channel. Every call the worker makes now takes the channel through _sidecar_stub, which only hands out the current connection, and a channel is closed only after it is retired and its count drops to zero. stop() waits up to five seconds for those calls and then closes the channel anyway; by then any such call has long started, so closing it only cancels it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> * fix(workflow): verify a connection before results use it, and bound every call on it Addresses review feedback on the reconnect fix: - Hello runs on the new connection before it is published, so a result waiting for a reconnect is never sent on a connection that then fails. - Result sends and the history fetch get a 30s deadline. A retired channel is closed only once no call is in flight on it, so these deadlines are what bound that close and stop(); the close itself never runs early, since that could land between a result and its oversized-result fallback. - stop() keeps the connection open for its drain, so a result that becomes ready while the worker stops is still delivered. - Releasing a call wakes waiters only when the channel's count reaches zero. - The run loop tracks a connected flag instead of an unused stub. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> --------- Signed-off-by: Fabian Martinez <46371672+famarting@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 9136b37) Signed-off-by: dapr-bot <dapr-bot@users.noreply.github.com>
dapr-bot
added
automerge
dapr-bot
enabled auto-merge (squash)
October 21, 2026 09:42
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release-1.19 #1324 +/- ##
===============================================
Coverage ? 87.15%
===============================================
Files ? 163
Lines ? 13066
Branches ? 0
===============================================
Hits ? 11388
Misses ? 1678
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
nelson-parente
approved these changes
Oct 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 9136b37 from #1312.