Report stats span collapses over telemetry (client-side stats)

[dougqh]

What Does This Do?

Report client-side trace-stats span collapses over telemetry as a new stats.collapsed_spans counter (namespace tracers), mirroring the existing statsd metric datadog.tracer.stats.collapsed_spans.

Motivation

Required by the cardinality limits & span-derived tags RFCs

The stats engine already counts every span collapse — cardinality-limit collapses, length ("oversized") collapses, and whole-key table drops — and emits them to statsd. But that path is invisible when no dogstatsd sink is configured: the aggregator's HealthMetrics is HealthMetrics.NO_OP in that case (CoreTracer.java:735), so the counts are simply dropped. Telemetry is always wired, so surfacing the same signal there makes cardinality-collapse visibility independent of statsd.

Additional Notes

Implementation Details

• New StatsMetrics holder (in internal-api, alongside BaggageMetrics): a per-reason AtomicLong counter in a ConcurrentHashMap, drained by CoreMetricCollector as a count metric — same pattern as the baggage counters.
• Fed directly at each collapse site (CoreHandlers, PeerTagSchema, AdditionalTagsSchema, and the whole-key drop in Aggregator), right next to the existing HealthMetrics call — deliberately not hooked inside TracerHealthMetrics, so it fires even when statsd is off.
• Tagged by collapse reason, matching the statsd tags: collapsed:<field>, collapsed:peer_tags, collapsed:additional_metric_tags, oversized:additional_metric_tags, collapsed:whole_key.

Cardinality safety

The per-reason map is dynamic, but the reason set is bounded by construction (9 core fields + peer_tags + the two additional-tags reasons + whole_key) — the cardinality limits this metric reports on are the very thing that bound it. So the map itself cannot blow up.

Testing

• StatsMetricsTest (JUnit 5): accumulation, drain-delta semantics, per-reason isolation, non-positive no-op.
• AdditionalTagsSchemaTest: a new case asserting the reset site feeds the telemetry counter (not just statsd).

---

Stacked on #11402 (client-side additional metric tags) — shares the schema/handler reset sites introduced there.

🤖 Generated with Claude Code

[dd-octo-sts]

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.96 s	13.94 s	[-0.8%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	12.93 s	13.05 s	[-1.7%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent	16.87 s	16.78 s	[-0.3%; +1.4%] (no difference)
startup:petclinic:iast:Agent	16.74 s	16.93 s	[-1.9%; -0.4%] (maybe better)
startup:petclinic:profiling:Agent	16.70 s	16.83 s	[-1.6%; +0.2%] (no difference)
startup:petclinic:sca:Agent	16.93 s	16.75 s	[+0.0%; +2.3%] (maybe worse)
startup:petclinic:tracing:Agent	15.59 s	16.18 s	[-7.7%; +0.3%] (no difference)

Commit: 3ee71215 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[dougqh]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

dd-trace-java/internal-api/src/main/java/datadog/trace/api/Config.java

Line 5273 in c1916aa

return tryMakeImmutableSet(configProvider.getList(TRACE_STATS_ADDITIONAL_TAGS));

Snapshot the additional-tags configuration

When DD_TRACE_STATS_ADDITIONAL_TAGS is enabled, this getter re-queries ConfigProvider instead of returning an immutable value captured by the Config constructor. Consequently, mutable sources such as system properties can change this setting after configuration initialization, and the new setting is also absent from Config.toString() startup diagnostics. Store it in a final field during construction and include it in the configuration summary, as required for newly added configurations.

AGENTS.md reference: AGENTS.md:L37-L44

---

dd-trace-java/internal-api/src/main/java/datadog/trace/api/telemetry/CoreMetricCollector.java

Line 73 in c1916aa

long value = counter.getValueAndReset();

Preserve collapse deltas when the telemetry queue is full

When metricsQueue is full, this calls getValueAndReset() before discovering that offer() fails. TaggedCounter advances its previousCount during that call, so the rejected collapse delta is permanently lost instead of being retried on the next telemetry cycle; this is especially likely for these counters because they are collected after span and baggage metrics. Check queue capacity before draining or restore/re-add a rejected delta.

---

dd-trace-java/dd-trace-core/src/main/java/datadog/trace/common/metrics/ClientStatsAggregator.java

Lines 138 to 139 in c1916aa

	config.getTraceStatsCardinalityLimit(
	"additional_tags", MetricCardinalityLimits.ADDITIONAL_TAG_VALUE),

Reject unsafe additional-tag cardinality limits

When additional tags are enabled and DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT is set above 1 << 29, this value is passed directly to TagCardinalityHandler, whose constructor throws IllegalArgumentException; large values below that guard can instead attempt multi-gigabyte array allocations. Because this happens while constructing ClientStatsAggregator, a malformed setting can prevent tracer startup rather than falling back to the documented default. Validate a practical upper bound before constructing the schema.

---

dd-trace-java/dd-trace-core/src/main/java/datadog/trace/common/metrics/ClientStatsAggregator.java

Lines 592 to 594 in c1916aa

	if (previousPeerTagSchema != null) {
	previousPeerTagSchema.resetHandlers(healthMetrics, cardinalityLimitReporter);
	previousPeerTagSchema = null;

Retain retired schemas until producers release them

If a producer reads the old peer-tag schema and is then paused for more than one reporting interval—for example during a long scheduling or GC pause—this reset discards previousPeerTagSchema before that producer queues its trace. When the delayed snapshots are eventually consumed, they update the retired schema's handlers, but no later cycle resets that schema, so any resulting peer-tag collapse counts are silently omitted from health metrics and the new telemetry counter. Retain retired schemas until delayed producers can no longer reference them, rather than for a fixed single cycle.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[datadog-datadog-us1-prod]

More details

The new collapse counters preserve per-reason accumulation and delta-drain behavior under repeated updates, including non-positive counts; the guarded collector path also leaves pending deltas intact when telemetry capacity is full. No diff-only behavioral regression was identified. Full Gradle tests could not start because this sandbox lacks the required Java 25 toolchain.

Was this helpful? React 👍 or 👎

_{📊 Validated against 5 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 545e3f0 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 545e3f03c9

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[dougqh]

Thanks @codex. Reconciling against the current branch — this review was posted against c1916aa269, before the rebase onto the updated #11402 base and the follow-up commits:

1. Snapshot the additional-tags configuration (Config) — ✅ Fixed on base PR Add span-derived primary tags (CSS v1.3.0) #11402: traceStatsAdditionalTags is now a final field resolved in the constructor, returned directly by the getter, and included in toString() (f5b083bd80).
2. Preserve collapse deltas when the telemetry queue is full (CoreMetricCollector) — ✅ Fixed on this branch: prepareMetrics() now checks remainingCapacity() before getValueAndReset() and stops the stats drain early, so a full queue no longer resets a delta it can't enqueue; untouched counters are collected on the next cycle (545e3f03c9). The same guard for the pre-existing span/baggage loops is split out into Guard core-metric telemetry drain against queue-full delta loss #12081.
3. Reject unsafe additional-tag cardinality limits (ClientStatsAggregator) — ✅ Fixed on base PR Add span-derived primary tags (CSS v1.3.0) #11402: DD_TRACE_STATS_ADDITIONAL_TAGS_CARDINALITY_LIMIT flows through Config.getTraceStatsCardinalityLimit(...), which now rejects values above MAX_TRACE_STATS_CARDINALITY_LIMIT (1<<16) and falls back to the default rather than allocating oversized arrays or throwing during startup (4188b43471).
4. Retain retired schemas until producers release them (ClientStatsAggregator) — Known limitation, accepted by design. The retired previousPeerTagSchema is held for one reset cycle; the failure mode needs a peer-tag schema change (config / feature-discovery driven, rare) to coincide with a producer paused for more than a full reporting interval (~10s), and it affects only block-telemetry counts, never trace data. Retaining schemas until "no producer can reference them" would require producer-side liveness tracking we don't want on the hot path. Treating this as a pre-existing config-update race, not a Report stats span collapses over telemetry (client-side stats) #12070 regression.

Items 1–3 are resolved; item 4 is a documented accepted limitation.

[chatgpt-codex-connector]

To use Codex here, create a Codex account and connect to github.

[bric3]

LGTM but I have a concern on the method names. The code explicitly mentions it's mirroring on the same signal, yet the method names are different.

Moreover the generated doc is still awkward, in particular in StatsMetrics.java.

[bric3]

question: Why method are named differently, this raises question when reading the code, yet, it seems the sgtats are emitted on the same "conditions".

I suggest aligning method names, unless there's a valid reason not to.

[dougqh]

Fair point — juxtaposed like this they do read a little oddly. The names differ on purpose: each mirrors the vocabulary of the sink it feeds, and the two sinks are named independently.

• HealthMetrics.onTagCardinalityBlocked(...) is one of HealthMetrics' onX event callbacks; that subsystem's vocabulary is "blocked" (the offending tag value is blocked/collapsed to the sentinel), and its statsd tags are collapsed:/oversized:.
• StatsMetrics.onCollapsedSpans(...) mirrors the telemetry metric it increments, stats.collapsed_spans — the collapse reason rides in the tag rather than the method name.

So both fire under the same condition because they report the same underlying event to two sinks, but renaming either to match the other would make that method name diverge from the name of the thing it actually reports. I'd rather keep each aligned to its own metric/event name.

Open to a different split if you feel strongly — but I don't think there's a single verb that reads naturally against both stats.collapsed_spans and the health *_blocked family at once.