Skip to content

fix(typescript): enable ADOT instrumentation for TypeScript agents - #1893

Open
jariy17 wants to merge 1 commit into
mainfrom
fix/adot-typescript-observability
Open

fix(typescript): enable ADOT instrumentation for TypeScript agents#1893
jariy17 wants to merge 1 commit into
mainfrom
fix/adot-typescript-observability

Conversation

@jariy17

@jariy17 jariy17 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Fixes #1892
Depends on aws/agentcore-l3-cdk-constructs#311 (CodeZip path)

Problem

agentcore deploy sets up no ADOT/OpenTelemetry instrumentation for TypeScript agents. Spans from any third-party instrumentation library are silently dropped and nothing appears in CloudWatch. Three independent gaps, each sufficient on its own:

  1. enableOtel hardcoded off for TypeScriptschema-mapper.ts:
    const enableOtel = !isMcp && config.language !== 'TypeScript';
  2. The TypeScript Dockerfile had no instrumentation branch at all. The Python one branches on {{#if enableOtel}}; the TS one ended unconditionally at CMD ["npx", "tsx", "main.ts"]. So setting instrumentation.enableOtel: true on a TS agent was a silent no-op — there was nothing for it to render — while the schema documents it as wrapping the entrypoint.
  3. No OTel SDK in either TS template. Strands carried only @opentelemetry/api (the no-op API surface, which discards spans unless a provider is registered); Vercel AI carried nothing.

This is a regression

580cd10"fix(templates): remove OTEL, session storage, and gateway from TS templates", merged in #981 — deleted the working implementation: both otel-register.ts files, their imports from main.ts, seven @opentelemetry/* dependencies, and it flipped schema-mapper.ts to exclude TypeScript. The OTel removal was one commit inside a broader TS template PR, so it isn't visible from the PR title.

Changes

  • schema-mapper.ts — stop excluding TypeScript from enableOtel
  • container/typescript/Dockerfile — add the {{#if enableOtel}} branch. Node has no opentelemetry-instrument equivalent, so it preloads the ADOT distro via NODE_OPTIONS rather than wrapping the entrypoint
  • both TS templates — add @aws/aws-distro-opentelemetry-node-autoinstrumentation
  • vercelai/base/main.ts — force-enable the Vercel AI SDK's own telemetry (see below)
  • dockerfile-render.test.ts — extend to cover TypeScript

The CodeZip path needs a different mechanism (no Dockerfile to hook) and lives in aws/agentcore-l3-cdk-constructs#311.

Why the Vercel template needs an explicit telemetry toggle

Turning on ADOT was necessary but not sufficient for Vercel AI. With instrumentation loaded and traces flowing, Vercel agents still produced only network-level spans — no model spans, no gen_ai.* attributes — while Strands produced a full GenAI tree from the same runtime.

Two compounding causes:

  • ADOT's Vercel instrumentation patches the ai module at import time. Both templates are "type": "module" and load instrumentation via --require (a CJS hook), which never observes an ESM import ... from 'ai'.
  • Under CodeZip it cannot work at all: esbuild inlines ai into the bundle, so there is no module resolution event left to intercept. An ESM loader hook would have fixed Container and silently missed CodeZip.

Setting experimental_telemetry on the call site sidesteps both. ai@6 resolves the global tracer (trace.getTracer('ai') from @opentelemetry/api, shared via globalThis), which ADOT registers at startup — so it works regardless of bundling or module system. Gated on AGENT_OBSERVABILITY_ENABLED === 'true', matching ADOT's own gate.

Strands needs no equivalent: its template self-instruments via @opentelemetry/api.

Verification

Deployed all four TypeScript agent shapes to a live account, invoked each 5 times, waited for ingestion, and queried X-Ray.

All four log AWS Distro of OpenTelemetry automatic instrumentation started successfully and produce X-Ray spans including downstream Bedrock calls.

Vercel AI, before vs after:

before:  SdkVercelZip.DEFAULT
           - bedrock-runtime.us-east-1.amazonaws.com:443
           - 169.254.169.254:80

after:   SdkVercelZip.DEFAULT
           - chat us.anthropic.claude-sonnet-4-5-20250929-v1:0
             - chat us.anthropic.claude-sonnet-4-5-...
               - bedrock-runtime.us-east-1.amazonaws.com:443

Attributes now present: gen_ai.request.model, gen_ai.response.model, gen_ai.response.finish_reasons, gen_ai.operation.name, gen_ai.provider.name, gen_ai.system. Confirmed on both CodeZip and Container.

Tests: asset suite green (141), including the new TS Dockerfile render cases.

Known cosmetic issue, not addressed here

Runtime logs show @aws/...-instrumentation-vercel-ai Failed to register VercelAISpanProcessor. It is harmless and unrelated: ADOT's setTracerProvider is called once before the real provider exists (logged at warn) and again after, when it succeeds (logged at debug, so invisible in production). It also appears for Strands agents, which have no ai package installed at all. Worth an upstream log-level fix on aws-observability/aws-otel-js-instrumentation; nothing is lost.

Follow-up

If LangChain or OpenAI-Agents TypeScript templates are added later, they will hit the same ESM/bundling wall — their ADOT instrumentation patches modules identically and will be equally inert. Each will need a per-SDK telemetry toggle, or an --import-based ESM loader hook on the Container path.

TypeScript agents received no OpenTelemetry instrumentation, so spans from any
third-party instrumentation library were silently dropped and nothing appeared
in CloudWatch. Three gaps, each sufficient on its own:

- `enableOtel` was hardcoded off for TypeScript in the render config
- the TypeScript Dockerfile had no instrumentation branch at all, so setting
  `instrumentation.enableOtel` on a TS agent was a silent no-op
- neither TS template depended on an OTel SDK — only the no-op
  `@opentelemetry/api`, which discards spans unless a provider is registered

This is a regression: 580cd10 ("remove OTEL, session storage, and gateway from
TS templates", #981) deleted the previous working implementation.

Node has no `opentelemetry-instrument` equivalent, so the Dockerfile preloads
the ADOT distro via NODE_OPTIONS instead of wrapping the entrypoint. The CodeZip
path is handled in @aws/agentcore-cdk.

Also force-enables the Vercel AI SDK's own telemetry. ADOT's Vercel
instrumentation patches the `ai` module at import time, which never fires under
CodeZip because esbuild inlines `ai` into the bundle, leaving no import to
intercept. Toggling the SDK's telemetry works regardless of bundling: ai@6 pulls
the global tracer that ADOT registers. Without this, Vercel agents produced only
network-level spans — no model spans, no gen_ai.* attributes. Strands is
unaffected because its template self-instruments via @opentelemetry/api.

Extends the Dockerfile render test to cover TypeScript; it previously only
exercised the Python Dockerfile, which is why this regressed unnoticed.

Verified on a live account: all four agent shapes (Strands/VercelAI x
CodeZip/Container) produce X-Ray spans, and both Vercel shapes now emit
`chat <model>` spans carrying gen_ai.request.model, gen_ai.response.model,
and gen_ai.response.finish_reasons.
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Package Tarball

aws-agentcore-0.25.0.tgz

How to install

gh release download pr-1893-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.25.0.tgz

@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 3, 2026
@github-actions github-actions Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 3, 2026
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 41.09% 15754 / 38332
🔵 Statements 40.35% 16802 / 41631
🔵 Functions 35.2% 2713 / 7706
🔵 Branches 34.26% 10535 / 30748
Generated in workflow #4446 for commit f0efe1e by the Vitest Coverage Report Action

@jariy17
jariy17 requested a review from aidandaly24 August 7, 2026 17:44
@jariy17 jariy17 closed this Sep 1, 2026
@jariy17 jariy17 reopened this Sep 1, 2026
@github-actions github-actions Bot added size/s PR size: S and removed size/s PR size: S labels Sep 1, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 1, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Changes requested

Good direction — flipping on enableOtel for TypeScript and preloading the ADOT distro in the Container Dockerfile is the right move, and the experimental_telemetry toggle in the Vercel AI template correctly sidesteps the CodeZip esbuild-inlining issue. Two gaps to address before this is complete, though:

1. Local agentcore dev won't emit any TS traces

In src/cli/operations/dev/codezip-dev-server.ts, the Node dev path spawns npx tsx watch main.ts with no NODE_OPTIONS. Since ADOT is not preloaded, no global tracer/meter provider is registered. The Vercel AI SDK's experimental_telemetry: { isEnabled: true } (driven by AGENT_OBSERVABILITY_ENABLED, which the CLI's collector sets) will silently drop spans onto the no-op tracer, and Strands' @opentelemetry/api calls will do the same. So running agentcore dev with the collector on will show zero traces for TS agents — which is exactly the workflow this PR advertises.

The Python dev path already handles this by prepending the OTEL sitecustomize.py dir to PYTHONPATH. TS needs the equivalent — e.g. when envVars.OTEL_EXPORTER_OTLP_ENDPOINT is set for the Node branch, add:

env.NODE_OPTIONS = [
  '--require @aws/aws-distro-opentelemetry-node-autoinstrumentation/register',
  env.NODE_OPTIONS,
].filter(Boolean).join(' ');

(and make sure node_modules is guaranteed to contain the ADOT package, which it will now that it's in the template package.json).

2. CodeZip TypeScript deploys will produce a broken entrypoint

In schema-mapper.ts the change from !isMcp && config.language !== 'TypeScript' to !isMcp unconditionally means TS CodeZip agents now persist instrumentation.enableOtel: true (via the schema default). The L3 construct at agentcore-l3-cdk-constructs/src/cdk/constructs/components/primitives/runtime/AgentCoreRuntime.ts L161 then wraps the entrypoint as ['opentelemetry-instrument', entrypointPath] — which is a Python console script that doesn't exist in a Node runtime. This is fine today only because AgentCore service reportedly doesn't yet accept NODE_* runtimes for CodeZip (per commit fa7500e in the constructs repo), but it's a landmine for when it does.

Options:

  • (Simplest, ship now) Restrict the toggle to Container: const enableOtel = !isMcp && (config.buildType === 'Container' || config.language !== 'TypeScript'); — matches what actually works.
  • Teach AgentCoreRuntime.ts to pick a Node-appropriate entrypoint (e.g. leave entryPoint alone and rely on a runtime-level env var / preload mechanism) when the runtime is NODE_*, and drop the language guard here.

Either is fine; the first is a one-liner and defers the CodeZip work to when the service is ready.

Nits (non-blocking, up to you)

  • @aws/aws-distro-opentelemetry-node-autoinstrumentation is added to dependencies in both TS templates unconditionally, but the Dockerfile preload is gated on enableOtel. If a user flips enableOtel: false they still install ~a lot of transitive deps. Not incorrect, just wasteful.

Tests look solid — real Handlebars rendering against the real template files, no mocking.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(typescript): agentcore deploy sets up no ADOT/OTel instrumentation for TypeScript agents, silently dropping all 3p telemetry

1 participant