Skip to content

fix(workflow): record runs that finish without a result so resume replays them - #7771

Open
AkshayShah03 wants to merge 2 commits into
google:mainfrom
AkshayShah03:fix/workflow-none-output-rerun-on-resume
Open

AkshayShah03 wants to merge 2 commits into
google:mainfrom
AkshayShah03:fix/workflow-none-output-rerun-on-resume

Conversation

@AkshayShah03

@AkshayShah03 AkshayShah03 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
A node run that finished with no output, route or transfer, and emitted no event of its own, left nothing in the session showing that it ran. On resume after a RequestInput, the scan found no recovered execution for it and ran it fresh, so its side effects (sending an email, writing a record) happened twice. The same applied to:

  • ctx.run_node children: check_interception Case 6 always reruns a dynamic run with no outcome;
  • children of a retried sub-workflow, which the retry guide documented as a limitation.

Solution:
The approach agreed on #7666: a typed completion marker, plus an explicit completion flag from the scan for dynamic runs.

  • NodeInfo.completed (new, optional): set on a content-less event that NodeRunner emits when a run finishes leaving no trace. That means no event of its own, no output, route or transfer, no error, no pending interrupts, not aborted, not stopped on a waiting child, and not a wait_for_output agent (task or chat mode) or rerun_on_resume node, since those rerun on resume to take new input. Any other wait_for_output run that ends waiting without emitting an event gets the marker too (added in d65ab2e, see Resuming a workflow re-runs the earlier no-output runs of a wait_for_output node, so its side effects happen twice #7905).
  • Nested Workflow nodes don't get the marker: they always rerun and replay their own children.
  • _reconstruct_node_states sets _ChildScanState.completed from a direct marker event and clears it if the run raises an interrupt afterwards.
  • check_interception Case 6 no longer reruns a dynamic run whose scan state is completed. Static runs already fast-forward once a recovered entry exists, which the marker now provides.

Runs that emit any event of their own keep today's behaviour, so agents and nodes that yield content are unaffected and add no extra events.

The marker is a persisted, caller-visible event, like the existing deltas-only flush events. It has no content, so is_final_response() is true for it. The only change for callers is one extra empty event per node run that finishes without leaving a trace.

Not covered here: a ctx.run_node child that emits content events but no output still reruns on resume. Telling "finished" apart from "stopped mid-run" for that case needs the marker on runs that emitted events too, which adds an event to many more runs, so I left it for a follow-up if you want it.

Docs: docs/guides/workflow/retry_config/index.md no longer says a retried sub-workflow reruns children that produced no output or state change, and the matching limitation is removed.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests:

  • test_workflow_hitl.py:

    • test_node_returning_none_is_not_rerun_on_resume (sequential)
    • test_parallel_node_returning_none_is_not_rerun_on_resume (parallel branch)
    • test_dynamic_child_returning_none_is_not_rerun_on_resume (ctx.run_node)

    Each is parametrized for resumable and non-resumable apps, and all 6 cases fail on main (['email', 'email']).

  • test_workflow_failures.py: test_retry_config_on_nested_workflow_replays_child_without_output. A no-output child of a retried sub-workflow runs once; on main it runs 3 times.

  • test_replay_interceptor.py: test_cross_turn_dynamic_run_recorded_completed_fast_forwards.

  • test_workflow_dynamic_nodes.py: removed the xfail from test_dynamic_node_none_output_not_rerun ("No completion marker event for None output"), which now passes.

$ pytest -n auto tests/unittests/workflow tests/unittests/runners tests/unittests/agents tests/unittests/flows tests/unittests/a2a tests/unittests/sessions tests/unittests/events
5316 passed, 58 skipped, 4 xfailed

$ pytest -n auto tests/unittests
21491 passed, 18 failed

The same 18 tests fail on main in my environment (integrations/vmaas/test_agent_engine_environment.py, the test_google_llm.py continuation-token tests that need a newer google-genai, and test_release_dependencies.py). I compared the failure lists, and they are identical.

pyink and isort report no changes.

Manual End-to-End (E2E) Tests:

  • The snippet from Workflow: a node that returns None runs again when the workflow resumes after RequestInput #7666: on main it prints after answering: ['email', 'email'], and with this change ['email'].
  • The same workflow with a fresh Runner per turn on DatabaseSessionService (SQLite) and SqliteSessionService: the email is sent once, and the stored history has one marker event, for wf@1/send_email@1.
  • A sub-workflow with RetryConfig(max_attempts=3) whose second node fails twice: the no-output first node runs 3 times on main and once with this change.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

@AkshayShah03

Copy link
Copy Markdown
Contributor Author

Rebased onto main (it had conflicted in _node_runner.py, where main's NodeInterruptedError handler now also copies the interrupt IDs; both lines are kept), and added d65ab2e as the follow-up discussed on #7905:

On its own, this change only stops the rerun once #7905's Case 6 change is in, since Case 6 still reruns every wait_for_output node until then. With both, the fuzzer goes from 24 replay divergences on main to 0. The workflow, runner, agent and flow suites pass, and the full suite shows the same failures as main.

@AkshayShah03
AkshayShah03 force-pushed the fix/workflow-none-output-rerun-on-resume branch from d65ab2e to a627716 Compare November 9, 2026 03:53
@AkshayShah03

Copy link
Copy Markdown
Contributor Author

Rebased onto main now that #7991 (ba142cf) is in. Only the test files conflicted, since both PRs added tests in the same places. With the Case 6 change on main, the marker from the follow-up commit now takes full effect, so I tightened the tests to check that end to end:

  • test_quiet_waiting_node_is_not_rerun_on_resume (new, resumable and not): a wait_for_output node that ended waiting without emitting an event runs once, not again on resume.
  • test_waiting_dynamic_child_keeps_parent_waiting_on_resume now also asserts the ctx.run_node child isn't run again, besides the parent staying waiting.

Both fail without a627716 and pass with it. The workflow, runner, agent and flow suites pass. With the same fuzzer as on #7905, main (which has #7991 now) still hits 24 replay divergences. This branch has 0 and adds no new failures.

…lays them

A node run that finished with no output, route or transfer and emitted no
event left nothing in the session showing it ran, so resuming after a
RequestInput ran it (and its side effects) again. The same applied to
ctx.run_node children and to children of a retried sub-workflow.

NodeRunner now emits a completion marker (NodeInfo.completed) for such a
run. The resume scan records it on the child's scan state, and
check_interception fast-forwards a dynamic run that has one instead of
running it again.

Fixes google#7666
The completion marker skipped every wait_for_output node, so a run that ended
WAITING without emitting any event left nothing in the session. On resume it
was treated as a new run and executed again with the input it already
handled.

Only agents (task or chat mode) and rerun_on_resume nodes rerun on resume to
take new input, so only they skip the marker now. Other wait_for_output nodes
record that their run ended. A ctx.run_node child recorded this way still
reports no output, so raise_on_wait keeps its parent WAITING.

See google#7905.
@AkshayShah03
AkshayShah03 force-pushed the fix/workflow-none-output-rerun-on-resume branch from a627716 to c88115d Compare November 11, 2026 16:28
@AkshayShah03

Copy link
Copy Markdown
Contributor Author

Hi @DeanChensj, a gentle check-in on this one. It's rebased on current main (c88115d) and mergeable again. Since it touches the same test file as other workflow fixes, it has needed a few rebases while waiting, so I wanted to ask whether anything would help get it reviewed: splitting it, or a different approach to the marker. #7991, which landed last week, is the other half of the #7905 fix, and with both in place the replay fuzzer from #7905 goes from 24 divergences to 0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow: a node that returns None runs again when the workflow resumes after RequestInput

2 participants