tracesSampler does not provide any useful information about the incoming HTTP request

[gajus]

For context, here is tracesSampler implementation:

    tracesSampler(samplingContext) {
      if (samplingContext.parentSampled !== undefined) {
        return samplingContext.parentSampled;
      }

      console.log('[tracesSampler] >>> ', { samplingContext });

      return 1;
    },

This logs:

[tracesSampler] >>>  {
  samplingContext: {
    transactionContext: { name: 'GET', parentSampled: undefined },
    parentSampled: undefined
  }
}

I've also instrumented http and fastify:

instrumentations: [
  getNodeAutoInstrumentations({
    '@opentelemetry/instrumentation-fastify': {
      requestHook: (span, info) => {
        console.log('[fastify] >>>', { span });
      },
    },
    // Disabling this to avoid hitting ENOENT errors
    '@opentelemetry/instrumentation-fs': {
      enabled: false,
    },
    '@opentelemetry/instrumentation-http': {
      requestHook: (span, request) => {
        console.log('[http] >>>', { span });
      },
    },
  }),
],

But these fire after tracesSampler. So how do I make a sampling decision when I see the request?

This is our entire implementation:

import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { AsyncLocalStorageContextManager } from '@opentelemetry/context-async-hooks';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-grpc';
import opentelemetry from '@opentelemetry/sdk-node';
import { RewriteFrames } from '@sentry/integrations';
import { init } from '@sentry/node';
import {
  getClient,
  SentryPropagator,
  SentrySampler,
  SentrySpanProcessor,
  setOpenTelemetryContextAsyncContextStrategy,
  setupEventContextTrace,
  setupGlobalHub,
  wrapContextManagerClass,
} from '@sentry/opentelemetry';
import { type Integration } from '@sentry/types';

// eslint-disable-next-line n/no-process-env
const { CONTRA_TRACING_ENABLED } = process.env as Record<string, string>;

export {
  type Breadcrumb,
  captureException,
  captureMessage,
  close,
  getCurrentHub,
  setUser,
  type Transaction,
  withScope,
} from '@sentry/node';
export { startInactiveSpan, startSpan } from '@sentry/opentelemetry';
export { type Extras } from '@sentry/types';

export const createSentry = ({
  integrations,
  root,
}: {
  integrations?: (integrations: Integration[]) => Integration[];
  root: string;
}) => {
  if (CONTRA_TRACING_ENABLED === 'true') {
    setupGlobalHub();
  }

  init({
    debug: false,
    instrumenter: 'otel',
    integrations: (defaultIntegrations) => {
      const customIntegrations = [
        ...defaultIntegrations.filter((defaultIntegration) => {
          return ![
            // Daniel implemented our own version of HttpContext.
            'HttpContext',
            // Not useful and uses a lot of vertical space on Sentry error reports.
            'Modules',
            // Associates wrong HTTP requests with the error.
            'Http',
            // Associates wrong log messages with the error.
            'Console',
          ].includes(defaultIntegration.name);
        }),
        new RewriteFrames({
          root,
        }),
      ];

      if (integrations) {
        return integrations(customIntegrations);
      }

      return customIntegrations;
    },
    normalizeDepth: 5,
    tracesSampler(samplingContext) {
      if (samplingContext.parentSampled !== undefined) {
        return samplingContext.parentSampled;
      }

      console.log('[tracesSampler] >>> ', { samplingContext });

      return 0;
    },
  });

  if (CONTRA_TRACING_ENABLED !== 'true') {
    return;
  }

  const client = getClient();

  if (!client) {
    return;
  }

  setupEventContextTrace(client);

  // You can wrap whatever local storage context manager you want to use
  const SentryContextManager = wrapContextManagerClass(
    AsyncLocalStorageContextManager,
  );

  const sdk = new opentelemetry.NodeSDK({
    contextManager: new SentryContextManager(),
    instrumentations: [
      getNodeAutoInstrumentations({
        '@opentelemetry/instrumentation-fastify': {
          requestHook: (span, info) => {
            console.log('[fastify] >>>', { span });
          },
        },
        // Disabling this to avoid hitting ENOENT errors
        '@opentelemetry/instrumentation-fs': {
          enabled: false,
        },
        '@opentelemetry/instrumentation-http': {
          requestHook: (span, request) => {
            console.log('[http] >>>', { span });
          },
        },
      }),
    ],
    sampler: new SentrySampler(client),
    spanProcessor: new SentrySpanProcessor(),
    textMapPropagator: new SentryPropagator(),
    traceExporter: new OTLPTraceExporter(),
  });

  // Ensure OpenTelemetry Context & Sentry Hub/Scope is synced
  setOpenTelemetryContextAsyncContextStrategy();

  sdk.start();
};

[gajus]

CC @lforst

[mydea]

Hey,

we have recently merged a PR to improve this here: #10426 - this is not yet released, but when the upcoming v7.100.0 release is out, this should get more data than before, including the name and the initial attributes. That should be enough to properly sample, hopefully!

[gajus]

@mydea Did something fundamentally change in v7.100 in how trace spans are propagated?

This used to work just fine:

app.get('/test-sentry-tracing', async () => {
  const activeContext = context.active();

  console.log('activeContext', activeContext);

  const activeSpan = trace.getSpan(activeContext);

  console.log('activeSpan', activeSpan);

  return 'OK';
});

But now it just outputs:

@opentelemetry/api: Registered a global for diag v1.7.0.
Applying patch for dns
patch lookup function
patch lookup function
Applying patch for net module
@opentelemetry/instrumentation-pg Applying patch for pg-pool@3.6.1
@opentelemetry/instrumentation-pg Applying patch for pg@8.11.3
@opentelemetry/instrumentation-pg Patching pg.Client.prototype.query
@opentelemetry/instrumentation-http Applying patch for https@21.6.0
@opentelemetry/instrumentation-http Applying patch for http@21.6.0
@opentelemetry/instrumentation-grpc Applying patch for @grpc/grpc-js@1.7.3
@opentelemetry/instrumentation-grpc patched gRPC server
@opentelemetry/instrumentation-grpc patching client
@opentelemetry/instrumentation-grpc patching client
@opentelemetry/instrumentation-grpc patching loadPackageDefinition
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
@opentelemetry/instrumentation-grpc patched makeClientStreamRequest on grpc client
Applying patch for dns/promises
patch lookup function
@opentelemetry/instrumentation-graphql Applying patch for graphql@16.8.1 parse
@opentelemetry/instrumentation-graphql Applying patch for graphql@16.8.1 validate
@opentelemetry/instrumentation-graphql Applying patch for graphql@16.8.1 execute
Applying patch for dataloader@2.2.2
Applying patch for pino@8.16.1
@opentelemetry/instrumentation-fastify Applying patch for fastify@4.24.3
@opentelemetry/instrumentation-fastify Patching fastify constructor function
@opentelemetry/instrumentation-fastify Patching fastify preHandler function
@opentelemetry/instrumentation-fastify Patching fastify server.addHook function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
wrap lookup callback function and starts span
executing wrapped lookup callback function
executing original lookup callback function
wrap lookup callback function and starts span
wrap lookup callback function and starts span
executing wrapped lookup callback function
executing original lookup callback function
executing wrapped lookup callback function
executing original lookup callback function
wrap lookup callback function and starts span
executing wrapped lookup callback function
executing original lookup callback function
@opentelemetry/instrumentation-fastify Patching fastify preHandler function
@opentelemetry/instrumentation-fastify Patching fastify server.addHook function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
@opentelemetry/instrumentation-fastify Patching fastify route.handler function
wrap lookup callback function and starts span
executing wrapped lookup callback function
executing original lookup callback function
@opentelemetry/instrumentation-http https instrumentation incomingRequest
@opentelemetry/instrumentation-fastify Patching fastify reply.send function
activeContext BaseContext {
  _currentContext: Map(0) {},
  getValue: [Function (anonymous)],
  setValue: [Function (anonymous)],
  deleteValue: [Function (anonymous)]
}
activeSpan undefined

[mydea]

Not on purpose! Could you open a new issue for this, in the best case with a reproduction? Our e2e test seems to still work, but we changed some internals so I can't rule out something unexpected broke!

[gajus]

Will do some digging to rule out any internal changes, including testing against older version of Sentry.

[gajus]

tracesSampler: (samplingContext) => {
  console.log({ samplingContext });

  return true;
},

tracesSampler is not even invoked. Not sure where to even start debugging this. Feels like I am overlooking something obvious. Downgrading Sentry to 7.99.0 did not help. So I don't think it is a recent breaking change.

[gajus]

@mydea #10577