[Reasoning Engine] New agent deployment consistently fails on 1st attempt with "failed to start and cannot serve traffic", but immediate retry succeeds

[Ganeshvenkatesanofficial]

Environment

• OS: macOS 26.6.2 (Client) / Linux (Debian 12 / Ubuntu 22.04 on GKE/Cloud Run deployment containers)
• Python: 3.11.16
• pip: 26.2.1
• google-cloud-aiplatform: 1.165.1
• agentplatform: 1.165.1
• google-adk: 2.8.0
• google-genai: 0.1.1
• Region: us-east4

Description

We are seeing a consistent intermittent deployment failure when creating Google ADK-based Agent Engine deployments using source_packages.

The first deployment attempt fails with:

code: 3
Reasoning Engine resource [...] failed to start and cannot serve traffic.

However, if we immediately retry the exact same deployment with the exact same source code and configuration, the second attempt succeeds with HTTP 200 OK.

This has reproduced consistently across 5+ independent agent deployments in the same GCP project and us-east4 region.

The behavior appears to be specific to the initial/cold deployment attempt rather than an application or dependency error, because the identical deployment succeeds immediately after the first failure.

Google's current Agent Platform troubleshooting documentation also documents this exact failed to start and cannot serve traffic behavior for first-time BYOC creation and notes that internal Agent Runtime IAM provisioning/propagation can cause the initial deployment to fail and that retrying the deployment can succeed.

Steps to reproduce

1. Configure an Agent Engine deployment using source_packages.

2. Use Google ADK and custom runtime dependencies such as:

   • google-cloud-aiplatform
   • google-adk
   • mcp
   • sse-starlette

3. Deploy using:

remote_agent = client.agent_engines.create(config=deploy_config)

4. Cloud Build successfully builds the image and pushes the container layers to Artifact Registry.
5. Approximately 20 seconds after Cloud Build reports DONE, the deployment fails with:

RuntimeError: Failed to create Agent Engine: {
    'code': 3,
    'message': 'Reasoning Engine resource [...] failed to start and cannot serve traffic.'
}

6. Immediately execute the same client.agent_engines.create(config=deploy_config) call again.
7. The second deployment succeeds with HTTP 200 OK.

Minimal configuration

import agentplatform

client = agentplatform.Client(
    project="your-project-id",
    location="us-east4",
    http_options={"api_version": "v1beta1"},
)

deploy_config = {
    "source_packages": ["my_agent_pkg", "my_runtime"],
    "entrypoint_module": "my_agent_pkg.agent",
    "entrypoint_object": "app",
    "requirements_file": "my_agent_pkg/requirements.txt",
    "agent_framework": "google-adk",
    "gcs_dir_name": "gs://my-staging-bucket/staging/",
    "display_name": "customer-support-agent",
    "description": "Multi-agent workflow using Google ADK",
    "identity_type": "AGENT_IDENTITY",
    "env_vars": {
        "GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY": "true",
        "GOOGLE_GENAI_ENABLE_TELEMETRY": "true",
    },
    "class_methods": [
        {
            "name": "query",
            "api_mode": "",
            "parameters": {
                "type": "object",
                "properties": {
                    "user_id": {"type": "string"},
                    "message": {"type": "string"},
                },
                "required": ["user_id", "message"],
            },
        }
    ],
}

Cloud Logging timeline

For failed deployment:

Reasoning Engine: 1106552900240277504

2026-09-25 06:49:28.645 UTC
CreateReasoningEngine API call initiated

2026-09-25 06:49:37.406 UTC
Cloud Build starts
Build: ad6c9de7-3473-4dea-999d-d5f6e7d0d154

2026-09-25 06:50:52.692 UTC
Cloud Build completes and container is pushed

2026-09-25 06:51:13.033 UTC
Deployment fails with code 3

The failure occurs approximately 20.34 seconds after Cloud Build completes.

Important observation

The failure is not deterministic.

The following sequence succeeds:

Attempt 1
Cloud Build → SUCCESS
Container pushed → SUCCESS
Reasoning Engine startup → FAILED

Attempt 2
Same source code
Same dependencies
Same configuration
Same region
Same project

→ SUCCESS

No application code, dependency versions, deployment configuration, or source package contents are changed between attempts.

We have reproduced this behavior across 5+ separate agent deployments.

Stack trace

Traceback (most recent call last):

  File "/code/src/api/v1/AGENT_DEPLOY/adk_deploy.py", line 154, in _deploy_and_save
    result = deploy_workflow_agent(...)

  File "/code/src/utils/adk_deploy/workflow_agent_builder.py", line 1580, in deploy_workflow_agent
    remote_agent = client.agent_engines.create(config=deploy_config)

  File "/usr/local/lib/python3.11/site-packages/agentplatform/_genai/agent_engines.py", line 2042, in create
    raise RuntimeError(f"Failed to create Agent Engine: {operation.error}")

RuntimeError: Failed to create Agent Engine: {
    'code': 3,
    'message': 'Reasoning Engine resource
    [projects/1068605802800/locations/us-east4/reasoningEngines/1106552900240277504]
    failed to start and cannot serve traffic.'
}

Expected behavior

The initial client.agent_engines.create() request should successfully create the Agent Engine when:

• Cloud Build succeeds.
• The container image is successfully pushed.
• The same source/configuration succeeds immediately when retried.
• There are no changes to the deployment between attempts.

Actual behavior

The first deployment intermittently fails with:

Reasoning Engine resource [...] failed to start and cannot serve traffic.

An immediate retry with the exact same configuration succeeds.

Possible cause

There are two observations worth investigating:

1. The failure happens consistently around 20 seconds after Cloud Build completes.
2. A subsequent deployment succeeds without any configuration or source-code changes.

This could indicate a transient Agent Runtime initialization/readiness condition during the first deployment, such as internal provisioning, image availability/caching, startup initialization, or IAM propagation.

The exact 20.34-second timing is particularly notable, although we have not established from the available logs that a fixed 20-second startup/readiness probe is definitively responsible.

Google's current troubleshooting documentation identifies a similar first-time BYOC failure mode where internal IAM provisioning takes time to complete and recommends retrying the deployment.

Questions

Could the Agent Runtime team confirm:

1. Is there an internal startup/readiness timeout that can cause this failure during the initial deployment?
2. Is the ~20-second interval after Cloud Build completion expected behavior or a known timeout?
3. Is the first-attempt failure related to internal IAM/service-agent provisioning or propagation?
4. Why does an identical deployment succeed immediately after the first attempt?
5. Is the retry behavior expected to be required for first-time/cold deployments?
6. If startup initialization is the cause, could the startup/readiness timeout be increased or made configurable for custom Agent Engine runtimes?

Proposed improvement

If this is confirmed to be a startup/readiness timeout rather than an IAM propagation issue, it would be useful to expose a configurable startup probe/readiness configuration for custom Agent Engine deployments, for example:

deploy_config = {
    ...
    "startup_probe": {
        "initial_delay_seconds": 30,
        "timeout_seconds": 90,
    }
}

Alternatively, Agent Runtime could internally allow a longer startup window for cold deployments involving large Python dependency sets.

For comparison, Google Cloud's Agent Platform documentation supports startup probes for workloads that require additional initialization time before health checks begin.

Workaround

Currently, the practical workaround is to retry the exact same deployment request after the first failure:

try:
    remote_agent = client.agent_engines.create(config=deploy_config)
except RuntimeError:
    remote_agent = client.agent_engines.create(config=deploy_config)

However, this is not ideal for production deployment automation because the first request creates a failed Reasoning Engine resource and the deployment process must rely on a second create attempt.

We would appreciate confirmation of the underlying cause and whether this behavior is expected or can be addressed in Agent Runtime/Agent Engine.