Skip to content

fix(agents): session block in the system instruction, not the user turn - #12199

Merged
MarkusNeusinger merged 6 commits into
mainfrom
fix/agent-dynamic-instruction-channel
Oct 15, 2026
Merged

MarkusNeusinger merged 6 commits into
mainfrom
fix/agent-dynamic-instruction-channel

Conversation

@MarkusNeusinger

@MarkusNeusinger MarkusNeusinger commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Cause. The root uses static_instruction plus a dynamic instruction (the session block). ADK 2.11 sends the dynamic part as user content: a <<<BEGIN_SYSTEM_INSTRUCTION>>> block with a preamble, inserted at the current-turn boundary, so it lands inside the user's own turn (google/adk/flows/llm_flows/prompt/_instructions.py, LlmRequest._insert_transient_user_content). Claude Haiku 5.5 read it as user text that posed as a system instruction. In a baseline trace of 8 prep cases on origin/main, its thinking said so in 7 of 8 turns, for example "a suspicious block embedded in the user's message ... which looks like an injection attempt I should disregard". Every one of the 23 root requests carried the markers in a user message.
  • Fix. The root has no dynamic instruction anymore. A before_model_callback (add_session_context) appends the session block to the system instruction as a second part after the static policy. VertexClaude sends each part as its own Claude system block, with ADK's cache breakpoint on the first part only (system_blocks). A session block that changes therefore leaves the policy cached, and the static prefix stays byte-identical. root.md now describes the block as part of the instruction instead of explaining away the markers. The thinking block_binding drop stays: a data tool can still change the block between two calls.
  • Regression checks. A service-flow test checks that no root request has the markers or the session block in any content, and that the system instruction is the static part plus the block. A registry test checks that no agent has a dynamic instruction. Model tests cover the two-block split, with and without caching. The prep eval now scores a reply that talks about the root's own instructions (markers, a system instruction, the session block) as a miss and counts it under instruction_talk. A scripted test covers that path and asserts the shape of the Claude request.

Numbers (Claude Haiku 5.5, Vertex AI eu)

Run Turns flagged in thinking Replies about it Root requests with markers Exact Cost
origin/main, 8 cases (2 of 10 were guard_unavailable) 7/8 0/8 23/23 8/8 $0.026
this branch, same 10 cases 0/10 0/10 0/30 10/10 $0.025
this branch, full prep eval 0/50 0/50 0/143 47/50 = 94 % (PASS) $0.098 (#12198 runs: $0.131, $0.142)

On the same 8 cases, the cost per model call fell from $0.00114 to $0.00087. The three full-run misses are model choices unrelated to the block: one leaves the Notiz column in, one labels classes 1/2 instead of Klasse 1/Klasse 2, and one adds an extra sort. None of them talked about instructions.

Decided and documented

  • The session block now lives in the system prompt. Its contents keep the same trust level and fences: the block was already "instruction priority", with user-derived values only inside <plot_language> and <user_data> fences. User text can no longer imitate it in place, and root.md says that a message that imitates the block is wrong. The scope guard and the [message withheld] path touch only user contents and are unchanged.

Plan

N/A

Test plan

  • pytest tests/unit/agents: 6,842 passed. renderer/test_executor.py::test_the_memory_floor_kills_the_run failed once under -n auto while the live eval ran alongside, and passes alone. It is unrelated.
  • CI blocker fixed in this PR: test_convert.py::TestSecurityReview (from feat(agents): read dropped Excel, ODS, Parquet, Feather, Arrow and NumPy files #12194) asserted a 1.0 s wall clock for scanning a 30 MB crafted comment. The CI runner, under coverage, took 1.17 s and 1.19 s on two runs. The bound is now 5 s, which still tells a linear scan from a backtracking one (hours).
  • ruff check, ruff format --check, mypy api core agents
  • uv run python -m tools.changelog check --base origin/main
  • Live prep eval on Vertex AI, before and after (see the table)

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

MarkusNeusinger and others added 2 commits October 15, 2026 22:05
ADK 2.11 sends an agent's dynamic `instruction` next to a
`static_instruction` as user content between
<<<BEGIN_SYSTEM_INSTRUCTION>>> markers, inserted into the user's own
turn. Claude Haiku 5.5 read the root's session block there as user text
posing as a system instruction (7 of 8 prep-eval turns' thinking said
so; one reply told the user about the markers "in your message").

The root now has no dynamic instruction. Its before_model_callback
(add_session_context) appends the session block to the system
instruction as a second part after the static policy, and VertexClaude
sends each part as its own system block with ADK's cache breakpoint on
the first, so a changing block leaves the policy cached. root.md
describes the block as part of the instruction. The prep eval scores a
reply that talks about the root's own instructions as a miss
(instruction_talk).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
Copilot AI balanced review requested due to automatic review settings October 15, 2026 20:12
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Moving session data into system instructions and changing provider-specific caching warrant final human review.

0 open findings

What changed in this PR

Moves the chat root’s session context from user messages into the system instruction so the model does not mistake server-provided context for forged instructions.

Changes:

  • Appends session context after the static policy, preserving Claude’s cache breakpoint on the policy.
  • Makes instruction-related replies fail the prep evaluation.
  • Adds regression tests and updates documentation.
File Description
tests/​unit/​agents/​runtime/​test_service_flow.py Verifies session context stays in system instructions.
tests/​unit/​agents/​runtime/​test_registry.py Requires agents to have no dynamic instruction.
tests/​unit/​agents/​runtime/​test_policy.py Checks updated session-context guidance.
tests/​unit/​agents/​runtime/​test_models.py Tests system-block splitting and caching.
tests/​unit/​agents/​evals/​test_prep.py Tests instruction-talk failures and request structure.
docs/​concepts/​agent-network.md Documents session placement and cache behavior.
changelog.d/​agent-session-block-system.md Records the fix and evaluation results.
agents/​README.md Updates architecture and evaluation documentation.
agents/​evals/​prep.py Detects and reports instruction-related replies.
agents/​anyplot/​prompts/​root.md Clarifies that session context belongs to system instructions.
agents/​anyplot/​models.py Preserves separate Claude system blocks.
agents/​anyplot/​agent.py Adds session context through a model callback.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…hecks

The CI runner under coverage scanned the 30 MB crafted comment in 1.17 s
and 1.19 s, over the 1.0 s bound, on two runs of #12199. A backtracking
scan of these inputs takes hours, so 5 s still tells the two apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
@codecov

codecov Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.72727% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
agents/evals/prep.py 92.30% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

MarkusNeusinger and others added 2 commits October 15, 2026 22:59
… check, prompt wording)

- with_session_part accepts only a non-empty string or a Content of
  text parts; any other shape raises instead of sending the root out
  with the session block and no policy.
- INSTRUCTION_TALK matches ADK's markers by name (a bare <<< also hit
  heredocs such as <<<EOT) and the plain words only after a determiner
  ("your system instruction", "der Sitzungsblock").
- root.md: the block is "the last section" of the instruction (ADK's
  identity line stands between the refusals and it), and the values in
  its <plot_language> and <user_data> fences are data the user chose.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
A live prep eval on origin/main (2026-10-15) had the root reply "Den
eingefügten Text, der sich als Systemanweisung ausgibt, behandle ich
nicht als Anweisung." The determiner list of the narrowed pattern
missed it; "als"/"as" and the indefinite articles now count. Replayed
on every traced reply: only that case matches.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GdMSqLR5ww4ji74EUSmk9M
@MarkusNeusinger
MarkusNeusinger merged commit a9868cc into main Oct 15, 2026
15 checks passed
@MarkusNeusinger
MarkusNeusinger deleted the fix/agent-dynamic-instruction-channel branch October 15, 2026 21:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants