From 1f7848cbc2fc816650e8a520cce972a21b060cbd Mon Sep 17 00:00:00 2001 From: Douglas Q Hawkins Date: Mon, 27 Jul 2026 12:35:58 -0400 Subject: [PATCH 1/4] Document the client-side-stats package design (package-info) Add a package-info.java for datadog.trace.common.metrics giving a two-minute mental model of the re-architected client-side-stats (CSS) pipeline: an at-a-glance ASCII flow diagram plus prose on the single-writer threading model, the per-interval aggregate/flush cycle, the bound->sentinel->report-per-cycle cardinality invariant, and the three cardinality-handler families. The per-class javadocs carry the mechanics; this is the top-level shape for a newcomer. Co-Authored-By: Claude Opus 4.8 --- .../trace/common/metrics/package-info.java | 80 +++++++++++++++++++ 1 file changed, 80 insertions(+) create mode 100644 dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java diff --git a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java new file mode 100644 index 00000000000..289801c574a --- /dev/null +++ b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java @@ -0,0 +1,80 @@ +/** + * Client-side trace statistics (CSS): the tracer computes per-interval aggregate stats -- hit + * counts, error counts, and latency histograms grouped by a tuple of span labels (resource, + * service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to the Datadog + * agent every reporting interval. This lets the backend show accurate metrics even when individual + * spans are sampled away. + * + *

At a glance

+ * + * 
{@code
+ * App / producer threads                        Single aggregator thread
+ * ----------------------                        ------------------------
+ *
+ * span finishes                                 (1) drain inbox
+ *      |                                              |
+ *      v                     MPSC inbox               v
+ * build immutable       (lock-free hand-off)     (2) canonicalize each label through its
+ * SpanSnapshot        ----------------------->       cardinality handler (Core / PeerTag /
+ *      |                                              AdditionalTags); overflow -> sentinel
+ *      v                                              |
+ * (no locks, no                                       v
+ *  mutable state)                               (3) fold into AggregateTable, keyed by the
+ *                                                    canonical labels (counts + histograms)
+ *                                                     |
+ *                                                     v  every ~10s (reporting interval)
+ *                                                (4) flush table via MetricWriter (msgpack
+ *                                                    or OTLP) -> agent, then clear
+ * }
+ * + *

Threading model (the load-bearing rule)

+ * + * Producer application threads do almost nothing: each captures an immutable {@link + * datadog.trace.common.metrics.SpanSnapshot} of the finished span and hands it off through a + * lock-free MPSC {@code inbox}. A single aggregator thread ({@link + * datadog.trace.common.metrics.Aggregator}) drains that inbox and owns all mutable state + * -- the {@link datadog.trace.common.metrics.AggregateTable} and every cardinality handler -- as + * its sole writer. None of that state is synchronized; correctness rests entirely on the + * single-writer invariant. Any cross-thread request that must mutate (e.g. clearing the table on + * downgrade) does not touch the state directly -- it is funneled onto the aggregator thread as a + * signal through the same inbox. If you add a mutation path, it goes through the inbox too. + * + *

The cycle

+ * + * The aggregator folds each drained snapshot into an {@link + * datadog.trace.common.metrics.AggregateEntry}, keyed by the canonical form of its labels, updating + * that entry's counters and histograms. Once per reporting interval (~10s) it flushes the whole + * table through a {@link datadog.trace.common.metrics.MetricWriter} -- msgpack ({@link + * datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- and clears it for the next + * interval. Buckets are per-interval deltas, not cumulative. + * + *

Cardinality is the thing that must stay bounded

+ * + * Grouping keys come from span/user data, so distinct values -- and therefore the series count, its + * memory, and its backend cost -- would grow without bound if left alone. Every label is + * canonicalized through a {@link datadog.trace.common.metrics.TagCardinalityHandler} that caps the + * number of distinct values per tag and collapses any overflow into a shared sentinel, so + * over-cardinality folds into one entry rather than exploding the table. The collapses are counted + * per cycle and reported (to {@link datadog.trace.core.monitor.HealthMetrics} and, rate-limited, + * through {@link datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is + * observable without the bookkeeping itself becoming unbounded. + * + *

Three handler families

+ * + * + * + * Each {@link datadog.trace.common.metrics.SpanSnapshot} captures its own schema reference at + * capture time, so producer and aggregator agree on tag indexing even if the current schema is + * swapped out between capture and consumption. + */ +package datadog.trace.common.metrics; From 549142193a90a57f6a83c7a8473b75c905fb0ab8 Mon Sep 17 00:00:00 2001 From: Douglas Q Hawkins Date: Tue, 28 Jul 2026 12:10:55 -0400 Subject: [PATCH 2/4] Address review: correct package-info inaccuracies - report() resets entries (retained for reuse; stale expunged over cycles), it does not clear the whole table each interval - OTLP flushes to the OTLP endpoint, not the agent; only msgpack goes to the agent - core string fields use PropertyCardinalityHandler, peer/additional tags use TagCardinalityHandler; primitive key fields are copied directly Co-Authored-By: Claude Opus 4.8 --- .../trace/common/metrics/package-info.java | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java index 289801c574a..de2c12cbc54 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java +++ b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java @@ -1,9 +1,9 @@ /** * Client-side trace statistics (CSS): the tracer computes per-interval aggregate stats -- hit * counts, error counts, and latency histograms grouped by a tuple of span labels (resource, - * service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to the Datadog - * agent every reporting interval. This lets the backend show accurate metrics even when individual - * spans are sampled away. + * service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to Datadog + * every reporting interval. This lets the backend show accurate metrics even when individual spans + * are sampled away. * *

At a glance

* @@ -23,8 +23,9 @@ * canonical labels (counts + histograms) * | * v every ~10s (reporting interval) - * (4) flush table via MetricWriter (msgpack - * or OTLP) -> agent, then clear + * (4) flush via MetricWriter -- msgpack to the + * agent, OTLP to the OTLP endpoint -- then + * reset entries for the next interval * } * *

Threading model (the load-bearing rule)

@@ -45,19 +46,25 @@ * datadog.trace.common.metrics.AggregateEntry}, keyed by the canonical form of its labels, updating * that entry's counters and histograms. Once per reporting interval (~10s) it flushes the whole * table through a {@link datadog.trace.common.metrics.MetricWriter} -- msgpack ({@link - * datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- and clears it for the next - * interval. Buckets are per-interval deltas, not cumulative. + * datadog.trace.common.metrics.SerializingMetricWriter}) or OTLP -- then resets each entry's + * counters for the next interval rather than discarding the table: entries (and their cross-cycle + * caches) are retained for reuse, and stale ones are expunged over subsequent cycles. Buckets are + * per-interval deltas, not cumulative. * *

Cardinality is the thing that must stay bounded

* * Grouping keys come from span/user data, so distinct values -- and therefore the series count, its - * memory, and its backend cost -- would grow without bound if left alone. Every label is - * canonicalized through a {@link datadog.trace.common.metrics.TagCardinalityHandler} that caps the - * number of distinct values per tag and collapses any overflow into a shared sentinel, so - * over-cardinality folds into one entry rather than exploding the table. The collapses are counted - * per cycle and reported (to {@link datadog.trace.core.monitor.HealthMetrics} and, rate-limited, - * through {@link datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is - * observable without the bookkeeping itself becoming unbounded. + * memory, and its backend cost -- would grow without bound if left alone. High-cardinality labels + * are canonicalized through a cardinality handler -- {@link + * datadog.trace.common.metrics.PropertyCardinalityHandler} for the core string fields (resource, + * service, operation, ...) and {@link datadog.trace.common.metrics.TagCardinalityHandler} for peer + * and additional tags -- that caps the number of distinct values per label and collapses any + * overflow into a shared sentinel, so over-cardinality folds into one entry rather than exploding + * the table. (Primitive key fields such as HTTP/gRPC status and the boolean flags are inherently + * low-cardinality and copied directly.) The collapses are counted per cycle and reported (to {@link + * datadog.trace.core.monitor.HealthMetrics} and, rate-limited, through {@link + * datadog.trace.common.metrics.CardinalityLimitReporter}) -- so the bounding is observable without + * the bookkeeping itself becoming unbounded. * *

Three handler families

* From c5c261fde9c02ce41178f425203b6a5889996969 Mon Sep 17 00:00:00 2001 From: Douglas Q Hawkins Date: Tue, 28 Jul 2026 13:33:54 -0400 Subject: [PATCH 3/4] Simplify package-info opening per review Adopt bric3's suggested intro: drop the CSS acronym and the span-label tuple in favor of a plainer sentence, and state the MetricWriter destinations (msgpack to the Datadog Agent, OTLP to the configured OTLP endpoint) up front. Co-Authored-By: Claude Opus 4.8 --- .../datadog/trace/common/metrics/package-info.java | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java index de2c12cbc54..b9b0bee1385 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java +++ b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java @@ -1,9 +1,11 @@ /** - * Client-side trace statistics (CSS): the tracer computes per-interval aggregate stats -- hit - * counts, error counts, and latency histograms grouped by a tuple of span labels (resource, - * service, operation, span kind, HTTP/gRPC status, peer tags, ...) -- and ships them to Datadog - * every reporting interval. This lets the backend show accurate metrics even when individual spans - * are sampled away. + * Client-side stats aggregate eligible completed spans by resource, service, operation, span kind, + * status, and configured tags. Each reporting interval produces hit and error counts and latency + * distributions. The aggregates are computed independently of trace sampling, so they include spans + * that are not sent individually. + * + *

A {@link datadog.trace.common.metrics.MetricWriter} publishes each interval: msgpack to the + * Datadog Agent or OTLP to the configured OTLP metrics endpoint. * *

At a glance

* From f720209d7696cd32c0871f265519820de032def0 Mon Sep 17 00:00:00 2001 From: Douglas Q Hawkins Date: Tue, 28 Jul 2026 13:34:49 -0400 Subject: [PATCH 4/4] Adopt reviewer's inbox-boundary diagram in package-info Replace the single-producer/aggregator sketch with bric3's suggested diagram: show the MPSC inbox as the boundary -- app/producer threads enqueue SpanSnapshots, control threads enqueue SignalItems -- and the aggregator thread as the sole owner handling both, enumerating the Report/Clear/Stop signals. Co-Authored-By: Claude Opus 4.8 --- .../trace/common/metrics/package-info.java | 41 +++++++++++-------- 1 file changed, 23 insertions(+), 18 deletions(-) diff --git a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java index b9b0bee1385..f248825e6be 100644 --- a/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java +++ b/dd-trace-core/src/main/java/datadog/trace/common/metrics/package-info.java @@ -10,24 +10,29 @@ *

At a glance

* * 
{@code
- * App / producer threads                        Single aggregator thread
- * ----------------------                        ------------------------
- *
- * span finishes                                 (1) drain inbox
- *      |                                              |
- *      v                     MPSC inbox               v
- * build immutable       (lock-free hand-off)     (2) canonicalize each label through its
- * SpanSnapshot        ----------------------->       cardinality handler (Core / PeerTag /
- *      |                                              AdditionalTags); overflow -> sentinel
- *      v                                              |
- * (no locks, no                                       v
- *  mutable state)                               (3) fold into AggregateTable, keyed by the
- *                                                    canonical labels (counts + histograms)
- *                                                     |
- *                                                     v  every ~10s (reporting interval)
- *                                                (4) flush via MetricWriter -- msgpack to the
- *                                                    agent, OTLP to the OTLP endpoint -- then
- *                                                    reset entries for the next interval
+ * App / Producer threads              Control threads
+ * ----------------------              ------------------------------
+ * eligible completed span             report schedule, tracer close,
+ *          |                          clear needed
+ * capture values and create                     |
+ * immutable SpanSnapshot                        |
+ *          |                                    |
+ *          +----------------+-------------------+
+ *                           v
+ *                   lock-free MPSC inbox
+ *                           |
+ *                           v
+ *                    Aggregator thread
+ *               (sole writer of aggregate state)
+ *                      /              \
+ *            SpanSnapshot              SignalItem
+ *                 |                        |
+ * canonicalize bounded fields   ReportSignal: reconcile peer schema,
+ * and tags; find/create entry                 expunge stale entries,
+ *                 |                           write interval deltas,
+ * update counts and histograms                reset active entries
+ *                                ClearSignal: clear the table
+ *                                 StopSignal: final report, then stop
  * }
* *

Threading model (the load-bearing rule)