Repository navigation
fix(workflow): record runs that finish without a result so resume replays them - #7771
AkshayShah03 wants to merge 2 commits into
Conversation
604d0cc to
d65ab2e
Compare
|
Rebased onto main (it had conflicted in
On its own, this change only stops the rerun once #7905's Case 6 change is in, since Case 6 still reruns every |
d65ab2e to
a627716
Compare
|
Rebased onto main now that #7991 (ba142cf) is in. Only the test files conflicted, since both PRs added tests in the same places. With the Case 6 change on main, the marker from the follow-up commit now takes full effect, so I tightened the tests to check that end to end:
Both fail without a627716 and pass with it. The workflow, runner, agent and flow suites pass. With the same fuzzer as on #7905, main (which has #7991 now) still hits 24 replay divergences. This branch has 0 and adds no new failures. |
…lays them A node run that finished with no output, route or transfer and emitted no event left nothing in the session showing it ran, so resuming after a RequestInput ran it (and its side effects) again. The same applied to ctx.run_node children and to children of a retried sub-workflow. NodeRunner now emits a completion marker (NodeInfo.completed) for such a run. The resume scan records it on the child's scan state, and check_interception fast-forwards a dynamic run that has one instead of running it again. Fixes google#7666
The completion marker skipped every wait_for_output node, so a run that ended WAITING without emitting any event left nothing in the session. On resume it was treated as a new run and executed again with the input it already handled. Only agents (task or chat mode) and rerun_on_resume nodes rerun on resume to take new input, so only they skip the marker now. Other wait_for_output nodes record that their run ended. A ctx.run_node child recorded this way still reports no output, so raise_on_wait keeps its parent WAITING. See google#7905.
a627716 to
c88115d
Compare
|
Hi @DeanChensj, a gentle check-in on this one. It's rebased on current main (c88115d) and mergeable again. Since it touches the same test file as other workflow fixes, it has needed a few rebases while waiting, so I wanted to ask whether anything would help get it reviewed: splitting it, or a different approach to the marker. #7991, which landed last week, is the other half of the #7905 fix, and with both in place the replay fuzzer from #7905 goes from 24 divergences to 0. |
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
A node run that finished with no output, route or transfer, and emitted no event of its own, left nothing in the session showing that it ran. On resume after a
RequestInput, the scan found no recovered execution for it and ran it fresh, so its side effects (sending an email, writing a record) happened twice. The same applied to:ctx.run_nodechildren:check_interceptionCase 6 always reruns a dynamic run with no outcome;Solution:
The approach agreed on #7666: a typed completion marker, plus an explicit completion flag from the scan for dynamic runs.
NodeInfo.completed(new, optional): set on a content-less event thatNodeRunneremits when a run finishes leaving no trace. That means no event of its own, no output, route or transfer, no error, no pending interrupts, not aborted, not stopped on a waiting child, and not await_for_outputagent (task or chat mode) orrerun_on_resumenode, since those rerun on resume to take new input. Any otherwait_for_outputrun that ends waiting without emitting an event gets the marker too (added in d65ab2e, see Resuming a workflow re-runs the earlier no-output runs of a wait_for_output node, so its side effects happen twice #7905).Workflownodes don't get the marker: they always rerun and replay their own children._reconstruct_node_statessets_ChildScanState.completedfrom a direct marker event and clears it if the run raises an interrupt afterwards.check_interceptionCase 6 no longer reruns a dynamic run whose scan state iscompleted. Static runs already fast-forward once a recovered entry exists, which the marker now provides.Runs that emit any event of their own keep today's behaviour, so agents and nodes that yield content are unaffected and add no extra events.
The marker is a persisted, caller-visible event, like the existing deltas-only flush events. It has no content, so
is_final_response()is true for it. The only change for callers is one extra empty event per node run that finishes without leaving a trace.Not covered here: a
ctx.run_nodechild that emits content events but no output still reruns on resume. Telling "finished" apart from "stopped mid-run" for that case needs the marker on runs that emitted events too, which adds an event to many more runs, so I left it for a follow-up if you want it.Docs:
docs/guides/workflow/retry_config/index.mdno longer says a retried sub-workflow reruns children that produced no output or state change, and the matching limitation is removed.Testing Plan
Unit Tests:
New tests:
test_workflow_hitl.py:test_node_returning_none_is_not_rerun_on_resume(sequential)test_parallel_node_returning_none_is_not_rerun_on_resume(parallel branch)test_dynamic_child_returning_none_is_not_rerun_on_resume(ctx.run_node)Each is parametrized for resumable and non-resumable apps, and all 6 cases fail on
main(['email', 'email']).test_workflow_failures.py:test_retry_config_on_nested_workflow_replays_child_without_output. A no-output child of a retried sub-workflow runs once; onmainit runs 3 times.test_replay_interceptor.py:test_cross_turn_dynamic_run_recorded_completed_fast_forwards.test_workflow_dynamic_nodes.py: removed thexfailfromtest_dynamic_node_none_output_not_rerun("No completion marker event for None output"), which now passes.The same 18 tests fail on
mainin my environment (integrations/vmaas/test_agent_engine_environment.py, thetest_google_llm.pycontinuation-token tests that need a newer google-genai, andtest_release_dependencies.py). I compared the failure lists, and they are identical.pyinkandisortreport no changes.Manual End-to-End (E2E) Tests:
mainit printsafter answering: ['email', 'email'], and with this change['email'].Runnerper turn onDatabaseSessionService(SQLite) andSqliteSessionService: the email is sent once, and the stored history has one marker event, forwf@1/send_email@1.RetryConfig(max_attempts=3)whose second node fails twice: the no-output first node runs 3 times onmainand once with this change.Checklist