Skip to content

Prevent ContinueAsNew from overriding termination - #1409

Open
wangbill (YunchuWang) wants to merge 3 commits into
mainfrom
yunchuwang-termination-race-regression-fix
Open

wangbill (YunchuWang) wants to merge 3 commits into
mainfrom
yunchuwang-termination-race-regression-fix

Conversation

@YunchuWang

@YunchuWang wangbill (YunchuWang) commented Sep 23, 2026 •

Copy link
Copy Markdown
Member

Prevent ContinueAsNew from overriding termination

Keep termination authoritative when a timer-fired event and a termination event arrive in the same work item. A pending ContinueAsNew currently bypasses the existing terminal guard, allowing the executor to return both Terminated and ContinuedAsNew. The dispatcher can then start another generation even though the runtime state retained the termination marker.

Changes

  • Move the existing terminal guard before ContinueAsNew selection in TaskOrchestrationContext.CompleteOrchestration; leave normal continuation, completion, and failure behavior unchanged.
  • Add deterministic tests for both event orders, full replay and reused-executor execution, and the real dispatcher. Assert one authoritative termination, an unchanged execution ID, and no additional execution or outbound continuation, with normal-behavior controls.

Validation

Local regression coverage

  • Before the fix, the new regression suite had six expected failures and five passing controls. After the fix, all 11 cases pass on both net8.0 and net48.
  • Debug preflight passed 69/69 selected Core tests on net8.0. After correcting the test path casing, all 11 regression cases were rerun successfully on both frameworks:
dotnet test .\test\DurableTask.Core.Tests\DurableTask.Core.Tests.csproj --configuration Debug --framework net8.0 --no-restore --filter "FullyQualifiedName~ContinueAsNewTerminationTests|FullyQualifiedName~ContinueAsNewTraceBehaviorTests|FullyQualifiedName~TaskOrchestrationContextTests|FullyQualifiedName~ExceptionHandlingIntegrationTests|FullyQualifiedName~DispatcherMiddlewareTests"
dotnet test .\test\DurableTask.Core.Tests\DurableTask.Core.Tests.csproj --configuration Debug --framework net48 --no-restore --filter FullyQualifiedName~ContinueAsNewTerminationTests

The earlier adjacent-test run on net48 had seven existing trace-activity failures. All seven were reproduced on the unpatched baseline, including when running the existing tracing tests alone; they are not changed here. Full repository/provider suites and hosted CI/CodeQL have not been run locally.

Bounded Azure validation of the proposed patch

Compared published Core 3.10.0 with private candidate 3.10.0-pr1409.e8ba6d46, built from this PR's e8ba6d46 commit, in disposable Azure Functions applications using in-process and .NET isolated execution. Candidate assembly hashes and source-commit metadata were verified in the in-process host and isolated worker; the isolated host's deployed candidate assembly was also verified. All non-Core package identities and hashes matched within each pair.

Execution model Formal trials per arm Control: continuation after termination Candidate: continuation after termination
In-process 36 1 0
.NET isolated 36 2 0

Direct coverage of a timer and termination in the same fetched work item came from one candidate in-process smoke case and two candidate isolated cases (one formal, one smoke). Each persisted exactly one Terminated completion, retained the execution ID, emitted no new ContinueAsNew, and remained terminated through an eight-second stability check and final inventory. The in-process candidate formal sweep did not produce a co-batch; its zero-loss result alone is not evidence of that path.

The patched cloud co-batches observed TimerFired before termination; the reverse order is covered by local regression tests.

Candidate formal runs had no conflicting-completion log, original duplicate-completion exception, or status reversion. Three candidate cases where ordinary completion won were not classified as continuation losses. All 20 no-termination controls completed after five normal continuations. All 152 formal and smoke instances reached terminal status.

These are bounded functional results, not failure-rate estimates or proof against every timing interleaving. The applications carried no customer traffic; this was a custom candidate validation, not a customer production deployment or a released package. Model-specific host runtimes and candidate/control compiler, signing, and release-build differences remain comparison limitations.

Related Issues

Related to Azure/azure-functions-durable-extension#2264.

Compatibility and rollout

This addresses the conflicting-completion path within one event batch, not every provider termination-timing scenario.

No package-version bump or downstream dependency update is included. The private candidate is not published to NuGet. For .NET isolated workers, the fixed Core version must also reach the worker SDK dependency that runs the executor; a host-only Core upgrade is insufficient, even though both were explicitly overridden for this validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a050d04d-d7f9-4d0f-8084-a2daa7000df4
Copilot AI lite review requested due to automatic review settings September 23, 2026 21:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Move the test file into the active lowercase project path so it is compiled and executed on case-sensitive checkouts.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
What changed in this PR

Updates orchestration completion so termination takes precedence over ContinueAsNew, with regression coverage for event ordering and dispatcher behavior.

Changes:

  • Prioritizes terminal completion over continuation.
  • Adds termination-race regression tests.
  • Test file is outside the active lowercase test project path.
File Summary
Test/​DurableTask.Core.Tests/​ContinueAsNewTerminationTests.cs Adds termination and continuation regression tests.
src/​DurableTask.Core/​TaskOrchestrationContext.cs Makes terminal completion authoritative.

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

Comment thread test/DurableTask.Core.Tests/ContinueAsNewTerminationTests.cs
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a050d04d-d7f9-4d0f-8084-a2daa7000df4
Copilot AI review requested due to automatic review settings September 23, 2026 22:21

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The reviewed changes and regression coverage address the termination precedence issue.

Review effort: Lite
Findings: None

Resolved since last review (1)

@YunchuWang wangbill (YunchuWang) added the ready for review Ready for review; not approval or confirmation that CI passed. label Sep 23, 2026
Copilot AI review requested due to automatic review settings September 24, 2026 17:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

No unresolved blocking issues were identified, and the supplied validation covers the changed behavior.

Review effort: Lite
Findings: None

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready for review Ready for review; not approval or confirmation that CI passed.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants