fix(typescript): enable ADOT instrumentation for TypeScript agents - #1893
fix(typescript): enable ADOT instrumentation for TypeScript agents#1893jariy17 wants to merge 1 commit into
Conversation
TypeScript agents received no OpenTelemetry instrumentation, so spans from any third-party instrumentation library were silently dropped and nothing appeared in CloudWatch. Three gaps, each sufficient on its own: - `enableOtel` was hardcoded off for TypeScript in the render config - the TypeScript Dockerfile had no instrumentation branch at all, so setting `instrumentation.enableOtel` on a TS agent was a silent no-op - neither TS template depended on an OTel SDK — only the no-op `@opentelemetry/api`, which discards spans unless a provider is registered This is a regression: 580cd10 ("remove OTEL, session storage, and gateway from TS templates", #981) deleted the previous working implementation. Node has no `opentelemetry-instrument` equivalent, so the Dockerfile preloads the ADOT distro via NODE_OPTIONS instead of wrapping the entrypoint. The CodeZip path is handled in @aws/agentcore-cdk. Also force-enables the Vercel AI SDK's own telemetry. ADOT's Vercel instrumentation patches the `ai` module at import time, which never fires under CodeZip because esbuild inlines `ai` into the bundle, leaving no import to intercept. Toggling the SDK's telemetry works regardless of bundling: ai@6 pulls the global tracer that ADOT registers. Without this, Vercel agents produced only network-level spans — no model spans, no gen_ai.* attributes. Strands is unaffected because its template self-instruments via @opentelemetry/api. Extends the Dockerfile render test to cover TypeScript; it previously only exercised the Python Dockerfile, which is why this regressed unnoticed. Verified on a live account: all four agent shapes (Strands/VercelAI x CodeZip/Container) produce X-Ray spans, and both Vercel shapes now emit `chat <model>` spans carrying gen_ai.request.model, gen_ai.response.model, and gen_ai.response.finish_reasons.
Package TarballHow to installgh release download pr-1893-tarball --repo aws/agentcore-cli --pattern "*.tgz" --dir /tmp/pr-tarball
npm install -g /tmp/pr-tarball/aws-agentcore-0.25.0.tgz |
|
Claude Security Review: no high-confidence findings. (run) |
Coverage Report
|
|
Claude Security Review: no high-confidence findings. (run) |
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Changes requested
Good direction — flipping on enableOtel for TypeScript and preloading the ADOT distro in the Container Dockerfile is the right move, and the experimental_telemetry toggle in the Vercel AI template correctly sidesteps the CodeZip esbuild-inlining issue. Two gaps to address before this is complete, though:
1. Local agentcore dev won't emit any TS traces
In src/cli/operations/dev/codezip-dev-server.ts, the Node dev path spawns npx tsx watch main.ts with no NODE_OPTIONS. Since ADOT is not preloaded, no global tracer/meter provider is registered. The Vercel AI SDK's experimental_telemetry: { isEnabled: true } (driven by AGENT_OBSERVABILITY_ENABLED, which the CLI's collector sets) will silently drop spans onto the no-op tracer, and Strands' @opentelemetry/api calls will do the same. So running agentcore dev with the collector on will show zero traces for TS agents — which is exactly the workflow this PR advertises.
The Python dev path already handles this by prepending the OTEL sitecustomize.py dir to PYTHONPATH. TS needs the equivalent — e.g. when envVars.OTEL_EXPORTER_OTLP_ENDPOINT is set for the Node branch, add:
env.NODE_OPTIONS = [
'--require @aws/aws-distro-opentelemetry-node-autoinstrumentation/register',
env.NODE_OPTIONS,
].filter(Boolean).join(' ');(and make sure node_modules is guaranteed to contain the ADOT package, which it will now that it's in the template package.json).
2. CodeZip TypeScript deploys will produce a broken entrypoint
In schema-mapper.ts the change from !isMcp && config.language !== 'TypeScript' to !isMcp unconditionally means TS CodeZip agents now persist instrumentation.enableOtel: true (via the schema default). The L3 construct at agentcore-l3-cdk-constructs/src/cdk/constructs/components/primitives/runtime/AgentCoreRuntime.ts L161 then wraps the entrypoint as ['opentelemetry-instrument', entrypointPath] — which is a Python console script that doesn't exist in a Node runtime. This is fine today only because AgentCore service reportedly doesn't yet accept NODE_* runtimes for CodeZip (per commit fa7500e in the constructs repo), but it's a landmine for when it does.
Options:
- (Simplest, ship now) Restrict the toggle to Container:
const enableOtel = !isMcp && (config.buildType === 'Container' || config.language !== 'TypeScript');— matches what actually works. - Teach
AgentCoreRuntime.tsto pick a Node-appropriate entrypoint (e.g. leaveentryPointalone and rely on a runtime-level env var / preload mechanism) when the runtime isNODE_*, and drop the language guard here.
Either is fine; the first is a one-liner and defers the CodeZip work to when the service is ready.
Nits (non-blocking, up to you)
@aws/aws-distro-opentelemetry-node-autoinstrumentationis added todependenciesin both TS templates unconditionally, but the Dockerfile preload is gated onenableOtel. If a user flipsenableOtel: falsethey still install ~a lot of transitive deps. Not incorrect, just wasteful.
Tests look solid — real Handlebars rendering against the real template files, no mocking.
Fixes #1892
Depends on aws/agentcore-l3-cdk-constructs#311 (CodeZip path)
Problem
agentcore deploysets up no ADOT/OpenTelemetry instrumentation for TypeScript agents. Spans from any third-party instrumentation library are silently dropped and nothing appears in CloudWatch. Three independent gaps, each sufficient on its own:enableOtelhardcoded off for TypeScript —schema-mapper.ts:{{#if enableOtel}}; the TS one ended unconditionally atCMD ["npx", "tsx", "main.ts"]. So settinginstrumentation.enableOtel: trueon a TS agent was a silent no-op — there was nothing for it to render — while the schema documents it as wrapping the entrypoint.@opentelemetry/api(the no-op API surface, which discards spans unless a provider is registered); Vercel AI carried nothing.This is a regression
580cd10 — "fix(templates): remove OTEL, session storage, and gateway from TS templates", merged in #981 — deleted the working implementation: both
otel-register.tsfiles, their imports frommain.ts, seven@opentelemetry/*dependencies, and it flippedschema-mapper.tsto exclude TypeScript. The OTel removal was one commit inside a broader TS template PR, so it isn't visible from the PR title.Changes
schema-mapper.ts— stop excluding TypeScript fromenableOtelcontainer/typescript/Dockerfile— add the{{#if enableOtel}}branch. Node has noopentelemetry-instrumentequivalent, so it preloads the ADOT distro viaNODE_OPTIONSrather than wrapping the entrypoint@aws/aws-distro-opentelemetry-node-autoinstrumentationvercelai/base/main.ts— force-enable the Vercel AI SDK's own telemetry (see below)dockerfile-render.test.ts— extend to cover TypeScriptThe CodeZip path needs a different mechanism (no Dockerfile to hook) and lives in aws/agentcore-l3-cdk-constructs#311.
Why the Vercel template needs an explicit telemetry toggle
Turning on ADOT was necessary but not sufficient for Vercel AI. With instrumentation loaded and traces flowing, Vercel agents still produced only network-level spans — no model spans, no
gen_ai.*attributes — while Strands produced a full GenAI tree from the same runtime.Two compounding causes:
aimodule at import time. Both templates are"type": "module"and load instrumentation via--require(a CJS hook), which never observes an ESMimport ... from 'ai'.aiinto the bundle, so there is no module resolution event left to intercept. An ESM loader hook would have fixed Container and silently missed CodeZip.Setting
experimental_telemetryon the call site sidesteps both.ai@6resolves the global tracer (trace.getTracer('ai')from@opentelemetry/api, shared viaglobalThis), which ADOT registers at startup — so it works regardless of bundling or module system. Gated onAGENT_OBSERVABILITY_ENABLED === 'true', matching ADOT's own gate.Strands needs no equivalent: its template self-instruments via
@opentelemetry/api.Verification
Deployed all four TypeScript agent shapes to a live account, invoked each 5 times, waited for ingestion, and queried X-Ray.
All four log
AWS Distro of OpenTelemetry automatic instrumentation started successfullyand produce X-Ray spans including downstream Bedrock calls.Vercel AI, before vs after:
Attributes now present:
gen_ai.request.model,gen_ai.response.model,gen_ai.response.finish_reasons,gen_ai.operation.name,gen_ai.provider.name,gen_ai.system. Confirmed on both CodeZip and Container.Tests: asset suite green (141), including the new TS Dockerfile render cases.
Known cosmetic issue, not addressed here
Runtime logs show
@aws/...-instrumentation-vercel-ai Failed to register VercelAISpanProcessor. It is harmless and unrelated: ADOT'ssetTracerProvideris called once before the real provider exists (logged atwarn) and again after, when it succeeds (logged atdebug, so invisible in production). It also appears for Strands agents, which have noaipackage installed at all. Worth an upstream log-level fix onaws-observability/aws-otel-js-instrumentation; nothing is lost.Follow-up
If LangChain or OpenAI-Agents TypeScript templates are added later, they will hit the same ESM/bundling wall — their ADOT instrumentation patches modules identically and will be equally inert. Each will need a per-SDK telemetry toggle, or an
--import-based ESM loader hook on the Container path.