Skip to content

fix(workflow): stop a resume failing on fenced JSON that passed live - #7972

Closed
anmolg1997 wants to merge 1 commit into
google:mainfrom
anmolg1997:fix/replay-strips-fenced-json-like-live
Closed

anmolg1997 wants to merge 1 commit into
google:mainfrom
anmolg1997:fix/replay-strips-fenced-json-like-live

Conversation

@anmolg1997

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:

A session paused on ADK 2.10 or earlier cannot be resumed on 2.11.0 or later when an LlmAgent node with an output_schema had replied with fenced JSON (```json ... ```). Those versions stored the node's reply as message text only, with output=None. On resume, _process_rehydrated_output rebuilds the output from that text without stripping the fence (_rehydration_utils.py#L194-L214), so the resumed run fails with WorkflowDataError on a reply that validate_schema accepted live.

Since 2.11.0 (6b96ba1, #6100) new sessions store the parsed output, so this only hits sessions stored by earlier versions.

Solution:

_process_rehydrated_output now calls _strip_json_code_fence, the helper validate_schema already uses, before it parses the text against the schema (_rehydration_utils.py#L198-L199). The live path and the replay path then share one fence helper.

Only the schema branch changes. Unfenced text comes back from the helper unchanged, and nodes with no schema or with output_schema=str still get the raw text.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.

New tests:

  • test_resume_replays_schema_agent_output_as_parsed_live in tests/unittests/workflow/test_workflow_hitl.py: a Workflow with a schema-typed LlmAgent on a scripted model, then a RequestInput pause. It runs to the pause, resumes, and checks that the next node gets the parsed dict and that the model was called once. It covers fenced and unfenced replies, resumable and non-resumable apps, and both storage shapes: the parsed output that 2.11+ stores, and the message-only event that 2.10 and earlier stored (message_only).
  • TestProcessRehydratedOutput::test_strips_markdown_fence_like_the_live_path in tests/unittests/workflow/utils/test_rehydration_utils.py.

The new tests against origin/main:

FAILED  test_resume_replays_schema_agent_output_as_parsed_live[fenced-message_only-False]
FAILED  test_resume_replays_schema_agent_output_as_parsed_live[fenced-message_only-True]
FAILED  TestProcessRehydratedOutput::test_strips_markdown_fence_like_the_live_path
PASSED  the other 6 cases (unfenced, or output_stored)
3 failed, 6 passed

With the fix: 9 passed.

$ pytest tests/unittests/workflow/utils/test_rehydration_utils.py tests/unittests/workflow/test_workflow_hitl.py \
    tests/unittests/workflow/utils/test_replay_interceptor.py tests/unittests/utils/test_schema_utils.py
238 passed

$ pytest tests/unittests -n auto            # Python 3.11.14, uv sync --all-extras
14 failed, 22886 passed, 96 skipped, 12 xfailed, 2 xpassed

$ tox -e py311,py312,py313,py314            # the four test files above
238 passed on each

The same 14 tests fail on unmodified main in that environment, for reasons outside this change: 13 in integrations/vmaas/test_agent_engine_environment.py (macOS stat has no -c), and test_release_dependencies.py::test_inclusive_upper_bounds_name_an_installed_release, because that environment resolved OpenTelemetry 1.42.1, below the release the declared bound names.

pre-commit run passes on the changed files. mypy ., compared with main the way CI does it, reports no new errors.

Manual End-to-End (E2E) Tests:

The script in #7971 pauses a workflow on one version and resumes it on another, with a SQLite DatabaseSessionService and a scripted model that returns fenced JSON. I paused the session on 2.10.0 and resumed copies of the same database:

### resume on google-adk==2.13.0 (latest on PyPI)
google.adk.workflow._errors.WorkflowDataError: Validation failed for rehydrated output against schema: 1 validation error for Digest
### resume on main 3565ad3 (unfixed)
google.adk.workflow._errors.WorkflowDataError: Validation failed for rehydrated output against schema: 1 validation error for Digest
### resume on this branch
approve got: {'title': 'Weekly', 'bullets': ['one']}
resumed output: {'digest': {'title': 'Weekly', 'bullets': ['one']}, 'approved': True}

A new session paused and resumed on the same version fails on 2.9.2 and 2.10.0 and passes on 2.11.0 and 2.13.0, which matches 6b96ba1.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules. (None.)

Additional context

I kept the change to the fence. Sending replay through validate_schema would also change how the output is dumped: live uses model_dump(exclude_none=True) (_schema_utils.py#L261) and replay uses model_dump() (_base_node.py#L176), so an unset optional field replays as an extra None key. That is a separate difference, and I left it alone.

When a session holds only the reply text of a finished LlmAgent node
with an output_schema, a resume rebuilds the node's output from that
text in _process_rehydrated_output. Sessions saved by ADK 2.10 and
earlier store schema replies this way. The function parsed the text
without stripping a Markdown code fence, so a fenced reply that
validate_schema accepted live failed the resumed run with
WorkflowDataError. Strip the fence there with the same helper
validate_schema uses.

Fixes google#7971
copybara-service Bot pushed a commit that referenced this pull request Nov 4, 2026
Merge #7972

Sessions stored by ADK 2.10 and earlier keep a schema agent's reply as
message text only, so resume rebuilds the output from that text. It now
strips a Markdown code fence first, as the live path does, so a fenced
reply accepted live no longer fails resume with WorkflowDataError.

Part of #7971

PiperOrigin-RevId: 1013369198
@adk-bot

adk-bot commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Thank you @anmolg1997 for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit f4fa73e.

Closing this PR as the changes are now in the main branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merged [Status] This PR is merged

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow paused on ADK 2.10 or earlier fails to resume when an output_schema agent's stored reply is fenced JSON

3 participants