safe_json_serialize dumps AuthCredential secret fields into span attributes; MCP headers are redacted, these are not

🔴 Required Information

Describe the Bug:

google.adk.telemetry._serialization.safe_json_serialize serializes pydantic
models with model_dump(mode="json"), which renders every field — including
the ones AuthCredential, OAuth2Auth, HttpCredentials and
ServiceAccountCredential declare as Field(repr=False).

adk_request_credential carries an AuthConfig as its arguments
(AuthToolArguments.auth_config) and receives one back as its function
response, and AuthConfig.exchanged_auth_credential is a full
AuthCredential. So the tool-call span for that flow exports the OAuth
access_token, refresh_token and client_secret in the clear, in
gcp.vertex.agent.tool_call_args and gcp.vertex.agent.tool_response.

This is the default configuration, not an opt-in:
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS defaults on
(TelemetryContext.should_add_content_to_legacy_spans). The spans go wherever
the exporter points — Cloud Trace, or any third-party OTel backend.

ADK already decides in three places that credentials must not reach an
exported span attribute:

telemetry/tracing.py, _build_llm_request_for_trace excludes
http_options.headers, extra_body and *client_args from the dumped
llm_request.config, with the comment "http_options carries
caller-supplied credentials … None of it may reach an exported span
attribute."
The MCP HTTP exchange record takes headers that "arrive already redacted,
so allowlisting a credential header yields the redaction marker rather
than the secret."
plugins/auto_tracing_helpers.py masks captured arguments through
_CREDENTIAL_TYPE_NAMES — which already names AuthConfig,
AuthCredential, AuthToolArguments and OAuth2Auth — plus a field-name
list and a bounded structural walk.

The rule and the list both already exist. They are applied in
AutoTracingPlugin's argument capture and not on the default tracing.py
path.

To be explicit about what is not being claimed: Field(repr=False) is a
repr control, and model_dump() rendering those fields is documented
pydantic behaviour, not a pydantic bug. The point is that ADK marks these
fields as not-for-display and maintains a redaction list naming their types,
and the telemetry serializer consults neither.

Steps to Reproduce:

pip install google-adk (or run from a checkout of main).
Save the script under Minimal Reproduction Code below as repro.py.
Run python repro.py with no ADK environment variables set.
It exits 1 and prints one LEAK line per secret that reached a span
attribute.

Expected Behavior:

No credential value reaches an exported span attribute. A trace may show
that a credential was present — the key, a redaction marker — but not its
value, consistent with what _build_llm_request_for_trace and the MCP header
record already do.

Observed Behavior:

LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_call_args' secret=access_token
LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_call_args' secret=refresh_token
LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_call_args' secret=client_secret
LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_response' secret=access_token
LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_response' secret=refresh_token
LEAK  span='execute_tool' attribute='gcp.vertex.agent.tool_response' secret=client_secret

repr() on the same object masks all three, which is the contrast worth
seeing:

access_token in repr(): False
refresh_token in repr(): False
client_secret in repr(): False
safe_json_serialize -> leaked: ['access_token', 'refresh_token', 'client_secret']

Environment Details:

ADK Library Version: 2.10.0 (reproduced from source at commit 044a1ec,
2026-09-25)
Desktop OS: Windows 11
Python Version: 3.12.3

Model Information:

Are you using LiteLLM: No
Which model is being used: N/A — the repro calls trace_tool_call directly
and needs no model.

🟡 Optional Information

Regression: Not a regression; safe_json_serialize has dumped models this
way for as long as it has existed.

Minimal Reproduction Code:

"""An OAuth access token lands in an exported ADK span attribute.

Default config: ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS defaults on.
Exits 1 if a secret reaches a span.
"""
import sys
from types import SimpleNamespace

from fastapi.openapi.models import OAuth2, OAuthFlows, OAuthFlowAuthorizationCode
from google.genai import types
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

from google.adk.auth.auth_credential import (
    AuthCredential, AuthCredentialTypes, OAuth2Auth)
from google.adk.auth.auth_tool import AuthConfig, AuthToolArguments
from google.adk.telemetry.tracing import trace_tool_call

TOKEN = "ya29.LEAKED-ACCESS-TOKEN"
REFRESH = "1//LEAKED-REFRESH-TOKEN"
SECRET = "LEAKED-CLIENT-SECRET"
SECRETS = {"access_token": TOKEN, "refresh_token": REFRESH,
           "client_secret": SECRET}

exporter = InMemorySpanExporter()
provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(exporter))
trace.set_tracer_provider(provider)

scheme = OAuth2(flows=OAuthFlows(authorizationCode=OAuthFlowAuthorizationCode(
    authorizationUrl="https://example.com/auth",
    tokenUrl="https://example.com/token", scopes={})))
cred = AuthCredential(
    auth_type=AuthCredentialTypes.OAUTH2,
    oauth2=OAuth2Auth(client_id="cid", client_secret=SECRET,
                      access_token=TOKEN, refresh_token=REFRESH))
auth_config = AuthConfig(auth_scheme=scheme, exchanged_auth_credential=cred)

# What ADK puts on the wire: adk_request_credential's args, and the client's
# auth response coming back as that call's function_response.
args = AuthToolArguments(function_call_id="fc-1",
                         auth_config=auth_config).model_dump(mode="json")
response_event = SimpleNamespace(
    id="ev-1",
    content=types.Content(role="user", parts=[types.Part(
        function_response=types.FunctionResponse(
            id="fc-1", name="adk_request_credential",
            response=auth_config.model_dump(mode="json")))]))
tool = SimpleNamespace(name="adk_request_credential",
                       description="request end user credentials",
                       custom_metadata=None)

tracer = provider.get_tracer("repro")
with tracer.start_as_current_span("execute_tool") as span:
    trace_tool_call(tool=tool, args=args,
                    function_response_event=response_event, span=span)

leaks = []
for s in exporter.get_finished_spans():
    for key, value in (s.attributes or {}).items():
        for name, secret in SECRETS.items():
            if secret in str(value):
                leaks.append((s.name, key, name))

for span_name, key, name in leaks:
    print(f"LEAK  span={span_name!r} attribute={key!r} secret={name}")
if not leaks:
    print("no secret reached any span attribute")
sys.exit(1 if leaks else 0)

How often has this issue occurred?: Always (100%)

Additional Context — suggested fix:

Apply the existing redaction rule at the serializer, reusing
auto_tracing_helpers' type and field-name lists rather than adding a second
set — which means lifting those lists into a module both sites can import.
Mask by declared credential type and by field name, keep the key so a trace
still shows a credential was present, and bound the walk so hitting a bound
elides rather than leaks.

I have this working with tests and no regressions against
tests/unittests/telemetry and tests/unittests/plugins, and can open a PR.

Disclosure: Reported to the Google VRP as issue 558689693; assessed as
below their security-tracking threshold, with public disclosure on GitHub
invited.

@chelsealong

Hi @chelsealong :) I am already pushing a PR

@IdoGol24

Hi @IdoGol24 , Thanks for reporting this and providing a thorough reproduction script.

Centralizing credential masking in the telemetry serializer makes sense and aligns with the existing redaction logic in auto_tracing_helpers. We will track and review the fix in #7312.

@IdoGol24

Hey @IdoGol24, nice find. Until #7312 lands, a mitigation for anyone on local or self-hosted OTel:

Set ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false (only false and 0 count as off; the default is on). Tool args and responses then go out as {}:

adk-python/src/google/adk/telemetry/context.py

Lines 132 to 137 in def458b

    
           def _read_add_content_to_legacy_spans() -> bool: 
        
             """Reads ``ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS`` (defaults on).""" 
        
             env_value = ( 
        
                 os.getenv(ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, 'true').strip().lower() 
        
             ) 
        
             return env_value not in _FALSY_ENV_VALUES

adk-python/src/google/adk/telemetry/tracing.py

Lines 311 to 347 in def458b

    
           if telemetry_config.should_add_content_to_legacy_spans: 
        
             span.set_attribute( 
        
                 "gcp.vertex.agent.tool_call_args", 
        
                 safe_json_serialize(args), 
        
             ) 
        
           else: 
        
             span.set_attribute("gcp.vertex.agent.tool_call_args", "{}") 
        
           # Tracing tool response 
        
           tool_call_id = "<not specified>" 
        
           tool_response = "<not specified>" 
        
           if ( 
        
               function_response_event is not None 
        
               and function_response_event.content is not None 
        
               and function_response_event.content.parts 
        
           ): 
        
             response_parts = function_response_event.content.parts 
        
             function_response = response_parts[0].function_response 
        
             if function_response is not None: 
        
               if function_response.id is not None: 
        
                 tool_call_id = function_response.id 
        
               if function_response.response is not None: 
        
                 tool_response = function_response.response 
        
           span.set_attribute(GEN_AI_TOOL_CALL_ID, tool_call_id) 
        
           if not isinstance(tool_response, dict): 
        
             tool_response = {"result": tool_response} 
        
           if function_response_event is not None: 
        
             span.set_attribute("gcp.vertex.agent.event_id", function_response_event.id) 
        
           if telemetry_config.should_add_content_to_legacy_spans: 
        
             span.set_attribute( 
        
                 "gcp.vertex.agent.tool_response", 
        
                 safe_json_serialize(tool_response), 
        
             ) 
        
           else: 
        
             span.set_attribute("gcp.vertex.agent.tool_response", "{}")

The trade-off is that you lose all tool args and responses from those spans, not only the credential ones.

Also, adk deploy ... --otel_to_cloud already sets it to false unless you override it, so those deploys aren't exposed by default:

adk-python/src/google/adk/cli/cli_deploy.py

Lines 1276 to 1277 in def458b

    
           if 'ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS' not in env_vars: 
        
             env_vars['ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS'] = 'false'

If you only need it off for some requests, capture_message_content on TelemetryContext overrides the env var per request:

adk-python/src/google/adk/telemetry/context.py

Lines 336 to 349 in def458b

    
             def should_add_content_to_legacy_spans(self) -> bool: 
        
               """Whether content goes on ADK-owned (legacy) spans. 
        
               Separate knob from the OTel content env var. A per-request 
        
               ``capture_message_content`` uses the OTel-spec span routing; otherwise this 
        
               falls back to ``ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS``, which defaults on. 
        
               """ 
        
               if ( 
        
                   not self._ignore_per_request 
        
                   and self.capture_message_content is not None 
        
               ): 
        
                 return _is_span_bearing(self.capture_message_content) 
        
               return self._env_add_content_to_legacy_spans

@sattyamjjain

Thanks @sattyamjjain, that's a useful stopgap. I ran it against def458b with the repro from this issue:

ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false does stop this leak: all three secrets disappear and gcp.vertex.agent.tool_call_args becomes {}. As you said, only false and 0 count; off still leaks.
The per-request override works in both directions. RunConfig.telemetry with capture_message_content=SPAN_ONLY turns legacy-span content back on even with the env var at false, and the credentials leak again (context.py#L336-L347). To make the env var stick, also set ADK_TELEMETRY_IGNORE_RUN_CONFIG=true (context.py#L181-L184); with both set, the override is ignored and nothing leaks.
The deploy default is narrower than it looks. It's set only in to_agent_engine, and only when --otel_to_cloud is passed explicitly (cli_deploy.py#L1268-L1287). If telemetry is enabled through GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY in .env, the else branch doesn't set it. adk deploy cloud_run and adk deploy gke pass --otel_to_cloud to the server, and I couldn't find anything on that path that sets it, so those deploys export content to Cloud Trace by default. (That one is from reading the code, not from a deploy.)
The trade-off is a bit wider than tool content: the same flag also blanks gcp.vertex.agent.llm_request, llm_response and gcp.vertex.agent.data (tracing.py#L639-L645).

#7312 masks only the credential material in the serializer, so the rest of the content survives. It still applies cleanly on def458b, and with it the repro passes under both the default config and the per-request override, leaving e.g. {"function_call_id": "fc-1", "auth_config": "<redacted>"}.

@sanketpatil06

hi @sanketpatil06 any update on this?
#7312 Still awits approvel, I have tried pinging a couple of times since.
Could you take a look?

Thanks for testing it and for the corrections. My deploy line was wrong. For anyone copying my earlier comment, here's the corrected stopgap:

Set both ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false and ADK_TELEMETRY_IGNORE_RUN_CONFIG=true. Without the second one, a RunConfig can turn content back on per request.
Only agent_engine deploys with an explicit --otel_to_cloud get the off default. cloud_run and gke deploys export content to Cloud Trace by default.
The flag also blanks the LLM request, response, and data attributes, not just tool content.

#7312 avoids all of that by masking only the credential fields, so +1 to getting it reviewed.

fix(sanitizer): mask CallToolResult content and structuredContent

fix(sanitizer): mask CallToolResult content and structuredContent (#412)

Navigation Menu

Repository navigation

safe_json_serialize dumps AuthCredential secret fields into span attributes; MCP headers are redacted, these are not #7311

🔴 Required Information

🟡 Optional Information

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def _read_add_content_to_legacy_spans() -> bool:
	"""Reads ``ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS`` (defaults on)."""
	env_value = (
	os.getenv(ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS, 'true').strip().lower()
	)
	return env_value not in _FALSY_ENV_VALUES

	if telemetry_config.should_add_content_to_legacy_spans:
	span.set_attribute(
	"gcp.vertex.agent.tool_call_args",
	safe_json_serialize(args),
	)
	else:
	span.set_attribute("gcp.vertex.agent.tool_call_args", "{}")

	# Tracing tool response
	tool_call_id = "<not specified>"
	tool_response = "<not specified>"
	if (
	function_response_event is not None
	and function_response_event.content is not None
	and function_response_event.content.parts
	):
	response_parts = function_response_event.content.parts
	function_response = response_parts[0].function_response
	if function_response is not None:
	if function_response.id is not None:
	tool_call_id = function_response.id
	if function_response.response is not None:
	tool_response = function_response.response

	span.set_attribute(GEN_AI_TOOL_CALL_ID, tool_call_id)

	if not isinstance(tool_response, dict):
	tool_response = {"result": tool_response}
	if function_response_event is not None:
	span.set_attribute("gcp.vertex.agent.event_id", function_response_event.id)
	if telemetry_config.should_add_content_to_legacy_spans:
	span.set_attribute(
	"gcp.vertex.agent.tool_response",
	safe_json_serialize(tool_response),
	)
	else:
	span.set_attribute("gcp.vertex.agent.tool_response", "{}")

	if 'ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS' not in env_vars:
	env_vars['ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS'] = 'false'

	def should_add_content_to_legacy_spans(self) -> bool:
	"""Whether content goes on ADK-owned (legacy) spans.

	Separate knob from the OTel content env var. A per-request
	``capture_message_content`` uses the OTel-spec span routing; otherwise this
	falls back to ``ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS``, which defaults on.
	"""
	if (
	not self._ignore_per_request
	and self.capture_message_content is not None
	):
	return _is_span_bearing(self.capture_message_content)
	return self._env_add_content_to_legacy_spans

safe_json_serialize dumps AuthCredential secret fields into span attributes; MCP headers are redacted, these are not #7311

Description

🔴 Required Information

🟡 Optional Information

Activity

IdoGol24 commented on Sep 27, 2026

sanketpatil06 commented on Sep 29, 2026

sattyamjjain commented on Sep 29, 2026

IdoGol24 commented on Sep 30, 2026

IdoGol24 commented on Oct 30, 2026

sattyamjjain commented on Oct 31, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions