From bb0e9cb86149899abfa07097eb25ea65dd294b0e Mon Sep 17 00:00:00 2001 From: becomeStar Date: Tue, 23 Dec 2025 00:28:57 +0900 Subject: [PATCH 1/4] A109: Target Attribute Filter for OpenTelemetry Metrics --- A109-target-attribute-filter.md | 170 ++++++++++++++++++++++++++++++++ 1 file changed, 170 insertions(+) create mode 100644 A109-target-attribute-filter.md diff --git a/A109-target-attribute-filter.md b/A109-target-attribute-filter.md new file mode 100644 index 000000000..6e52887fd --- /dev/null +++ b/A109-target-attribute-filter.md @@ -0,0 +1,170 @@ +A109: Target Attribute Filter for OpenTelemetry Metrics +---- +* Author(s): [becomeStar](https://github.com/becomeStar) +* Approver: a11r +* Status: Draft +* Implemented in: C++, Java (Java implementation will follow) +* Last updated: 2025-12-22 +* Discussion at: (to be filled after discussion thread is created) + +## Abstract + +Add an optional filter to control how the `grpc.target` attribute is recorded in +OpenTelemetry metrics, allowing rejected targets to be mapped to `"other"` to +reduce metric cardinality, while preserving existing behavior by default. + +## Background + +[gRFC A66][]'s per-call metrics include the `grpc.target` attribute, which can +have very high cardinality in large-scale deployments where clients connect to +many different server targets. This high cardinality can cause OpenTelemetry SDK +warnings (see [issue #12322](https://github.com/grpc/grpc-java/issues/12322)) +when the maximum allowed cardinality (default 2000, warning at 1999+) is +exceeded for instruments such as: + +* `grpc.client.attempt.started` +* `grpc.client.attempt.duration` +* `grpc.client.attempt.sent_total_compressed_message_size` +* `grpc.client.attempt.rcvd_total_compressed_message_size` +* `grpc.client.call.duration` + +A workaround exists using OpenTelemetry Views with `setAttributeFilter()` to +discard `grpc.target` entirely, but this is an all-or-nothing approach and not +a suitable replacement for selective filtering. + +[gRFC A66]: A66-otel-stats.md + +### Related Proposals +* [gRFC A66][]: OpenTelemetry Metrics + +## Proposal + +gRPC will add an API for applications to provide a filter function that +determines whether a target should be recorded as-is or mapped to `"other"` for +the `grpc.target` attribute in OpenTelemetry metrics. When no filter is +provided (default), all targets use their original target string, preserving +existing behavior. + +The string `"other"` is chosen as a stable, low-cardinality placeholder value +to represent all filtered targets. This value is intentionally fixed to ensure +consistent aggregation behavior across SDKs and deployments. The placeholder +value is not configurable to avoid further cardinality growth. + +The filtering applies to all client-side per-call instruments that include the +`grpc.target` attribute, including those defined in [gRFC A66][] and [gRFC +A96][]. + +[gRFC A96]: A96-retry-otel-stats.md + +### C++ + +gRPC C++ will add a method `SetTargetAttributeFilter` to +`OpenTelemetryPluginBuilder` that accepts an +`absl::AnyInvocable`. + +```cpp +OpenTelemetryPluginBuilder& SetTargetAttributeFilter( + absl::AnyInvocable + target_attribute_filter); +``` + +The filter is stored in the plugin state and applied when creating +`OpenTelemetryClientFilter`. If no filter is registered or if the filter returns +`true`, the original target string is used. Otherwise, `"other"` is used. + + +### Java + +gRPC Java will add a new method `targetAttributeFilter` to +`GrpcOpenTelemetry.Builder` that accepts a `Predicate`. The filter +defaults to `null` when unset, meaning all targets are recorded as-is. + +```java +public Builder targetAttributeFilter(@Nullable Predicate filter) +```` + +When a filter is provided, `filter.test(target)` is called when the client interceptor is created +for a channel. If the predicate returns `true`, the original target string is used as +`grpc.target`. If it returns `false`, the string `"other"` is used instead. + +The filter is applied when the client interceptor is created, meaning the +filtered target value is determined once per channel and reused for all calls +on that channel. + +## Rationale + +The `targetAttributeFilter` controls how the `grpc.target` attribute is recorded in OpenTelemetry metrics. Targets accepted by the filter are recorded as-is; rejected targets are replaced with `"other"` to limit metric cardinality. + +This approach is already implemented in gRPC C++, and bringing it to gRPC Java ensures consistent metric semantics across languages, which is important for multi-language deployments. + +**Alternative approach considered:** + +* Configuring a View with `setAttributeFilter()` to discard `grpc.target`. + * This is an all-or-nothing approach and only serves as a temporary workaround. + +**Reason for selection:** + +* Simple and straightforward to implement. +* Immediately addresses high-cardinality metrics. +* Aligns Java behavior with existing C++ implementation. + +## Implementation + +### C++ + +The implementation adds `SetTargetAttributeFilter` to +`OpenTelemetryPluginBuilder`: + +```cpp +OpenTelemetryPluginBuilder& +OpenTelemetryPluginBuilder::SetTargetAttributeFilter( + absl::AnyInvocable + target_attribute_filter) { + target_attribute_filter_ = std::move(target_attribute_filter); + return *this; +} +``` + +The filter is stored in the plugin state and applied when creating +`OpenTelemetryClientFilter`: + +```cpp +absl::StatusOr OpenTelemetryClientFilter::Create( + const grpc_core::ChannelArgs& args, ChannelFilter::Args /*filter_args*/) { + std::string target = + args.GetOwnedString(GRPC_ARG_SERVER_URI).value_or(""); + + if (OTelPluginState().target_attribute_filter == nullptr || + OTelPluginState().target_attribute_filter(target)) { + return OpenTelemetryClientFilter(std::move(target)); + } + return OpenTelemetryClientFilter("other"); +} +``` + +### Java + +The implementation adds `targetAttributeFilter` to +`GrpcOpenTelemetry.Builder`: + +```java +public Builder targetAttributeFilter(@Nullable Predicate filter) { + this.targetAttributeFilter = filter; + return this; +} +``` + +The filter is passed to `OpenTelemetryMetricsModule` and applied when recording +the target: + +```java +String recordTarget(String target) { + if (targetAttributeFilter == null) { + return target; + } + return targetAttributeFilter.test(target) ? target : "other"; +} +``` + +The filtered target is determined when the client interceptor is created and is +used for all metrics that include the `grpc.target` attribute for that channel. From f05b079bb7deea71a00e4e74a9c41e4f2f8eee86 Mon Sep 17 00:00:00 2001 From: becomeStar Date: Wed, 31 Dec 2025 17:17:11 +0900 Subject: [PATCH 2/4] A109: Define null handling behavior for target filter --- A109-target-attribute-filter.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/A109-target-attribute-filter.md b/A109-target-attribute-filter.md index 6e52887fd..0f192cf71 100644 --- a/A109-target-attribute-filter.md +++ b/A109-target-attribute-filter.md @@ -159,7 +159,7 @@ the target: ```java String recordTarget(String target) { - if (targetAttributeFilter == null) { + if (targetAttributeFilter == null || target == null) { return target; } return targetAttributeFilter.test(target) ? target : "other"; From 86c8007d123fdb510a8038e6e5a49511757a9fea Mon Sep 17 00:00:00 2001 From: becomeStar Date: Sat, 3 Jan 2026 17:29:36 +0900 Subject: [PATCH 3/4] A109: Use internal interface in Java to ensure Android compatibility --- A109-target-attribute-filter.md | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/A109-target-attribute-filter.md b/A109-target-attribute-filter.md index 0f192cf71..1dcf4d7b4 100644 --- a/A109-target-attribute-filter.md +++ b/A109-target-attribute-filter.md @@ -76,12 +76,14 @@ The filter is stored in the plugin state and applied when creating ### Java gRPC Java will add a new method `targetAttributeFilter` to -`GrpcOpenTelemetry.Builder` that accepts a `Predicate`. The filter -defaults to `null` when unset, meaning all targets are recorded as-is. +`GrpcOpenTelemetry.Builder` that accepts a `Predicate`. To ensure compatibility with Android +API levels < 24 (where `Predicate` is not available), the filter is converted and stored internally +using a package-private interface. +The filter defaults to `null` when unset, meaning all targets are recorded as-is. ```java public Builder targetAttributeFilter(@Nullable Predicate filter) -```` +``` When a filter is provided, `filter.test(target)` is called when the client interceptor is created for a channel. If the predicate returns `true`, the original target string is used as @@ -144,12 +146,25 @@ absl::StatusOr OpenTelemetryClientFilter::Create( ### Java -The implementation adds `targetAttributeFilter` to -`GrpcOpenTelemetry.Builder`: +The implementation adds `targetAttributeFilter` to `GrpcOpenTelemetry.Builder` and uses an internal +interface for storage. ```java +interface TargetFilter { + boolean test(String target); +} + public Builder targetAttributeFilter(@Nullable Predicate filter) { - this.targetAttributeFilter = filter; + if (filter == null) { + this.targetFilter = null; + } else { + this.targetFilter = new TargetFilter() { + @Override + public boolean test(String target) { + return filter.test(target); + } + }; + } return this; } ``` From cceb8746327539fa316115f661d0632a06142219 Mon Sep 17 00:00:00 2001 From: becomeStar Date: Tue, 7 Jul 2026 23:48:39 +0900 Subject: [PATCH 4/4] A109: clarify Java target filter scope --- A109-target-attribute-filter.md | 146 +++++++++++++++----------------- 1 file changed, 69 insertions(+), 77 deletions(-) diff --git a/A109-target-attribute-filter.md b/A109-target-attribute-filter.md index 1dcf4d7b4..b3ac897df 100644 --- a/A109-target-attribute-filter.md +++ b/A109-target-attribute-filter.md @@ -2,16 +2,17 @@ A109: Target Attribute Filter for OpenTelemetry Metrics ---- * Author(s): [becomeStar](https://github.com/becomeStar) * Approver: a11r -* Status: Draft -* Implemented in: C++, Java (Java implementation will follow) -* Last updated: 2025-12-22 -* Discussion at: (to be filled after discussion thread is created) +* Status: In Review +* Implemented in: Java ([grpc/grpc-java#12587][]) +* Last updated: 2026-07-07 +* Discussion at: (to be filled after the grpc-io notification is sent) ## Abstract -Add an optional filter to control how the `grpc.target` attribute is recorded in -OpenTelemetry metrics, allowing rejected targets to be mapped to `"other"` to -reduce metric cardinality, while preserving existing behavior by default. +Add an optional Java API to control how the `grpc.target` attribute is recorded +in gRPC Java OpenTelemetry client metrics, allowing rejected targets to be +mapped to `"other"` to reduce metric cardinality, while preserving existing +behavior by default. ## Background @@ -32,17 +33,48 @@ A workaround exists using OpenTelemetry Views with `setAttributeFilter()` to discard `grpc.target` entirely, but this is an all-or-nothing approach and not a suitable replacement for selective filtering. +A66 left overriding recorded target names as possible future work, and +[grpc/proposal#431][] later removed the target attribute filter requirement +from A66 because the requirements had not yet arisen. [grpc/grpc-java#12322][] +is such a requirement: Java users can exceed OpenTelemetry cardinality limits +when clients connect to thousands of targets. + +The Java implementation was reviewed and merged in [grpc/grpc-java#12587][] on +2026-01-09. + +### Related History: C++ target filtering + +gRPC C++ previously added a similar API in [grpc/grpc#34285][] based on the +original A66 proposal, then marked it deprecated in [grpc/grpc#36567][] after +[grpc/proposal#431][] removed the target attribute filter requirement from A66. +This proposal treats the C++ API as related history rather than proposing +broader cross-language behavior. + +The current C++ public header says that the filtering only applies to per-call +metrics and does not filter `grpc.target` on load-balancing policy metrics such +as those defined by [gRFC A78][]. In other words, this proposal intentionally +does not define a general mechanism for rewriting every `grpc.target` attribute +emitted by gRPC. + [gRFC A66]: A66-otel-stats.md +[grpc/grpc-java#12322]: https://github.com/grpc/grpc-java/issues/12322 +[grpc/grpc-java#12587]: https://github.com/grpc/grpc-java/pull/12587 +[grpc/grpc#34285]: https://github.com/grpc/grpc/pull/34285 +[grpc/grpc#36567]: https://github.com/grpc/grpc/pull/36567 +[grpc/proposal#431]: https://github.com/grpc/proposal/pull/431 +[gRFC A78]: A78-grpc-metrics-wrr-pf-xds.md ### Related Proposals * [gRFC A66][]: OpenTelemetry Metrics +* [gRFC A78][]: OpenTelemetry Metrics for WRR, Pick First, and XdsClient +* [gRFC A96][]: OpenTelemetry Metrics for Retries ## Proposal -gRPC will add an API for applications to provide a filter function that +gRPC Java will add an API for applications to provide a filter function that determines whether a target should be recorded as-is or mapped to `"other"` for -the `grpc.target` attribute in OpenTelemetry metrics. When no filter is -provided (default), all targets use their original target string, preserving +the `grpc.target` attribute in Java OpenTelemetry client metrics. When no filter +is provided (default), all targets use their original target string, preserving existing behavior. The string `"other"` is chosen as a stable, low-cardinality placeholder value @@ -50,43 +82,31 @@ to represent all filtered targets. This value is intentionally fixed to ensure consistent aggregation behavior across SDKs and deployments. The placeholder value is not configurable to avoid further cardinality growth. -The filtering applies to all client-side per-call instruments that include the -`grpc.target` attribute, including those defined in [gRFC A66][] and [gRFC -A96][]. +This proposal is limited to Java client-side OpenTelemetry metrics emitted by +gRPC Java's OpenTelemetry module that include the `grpc.target` attribute, +including the relevant metrics defined in [gRFC A66][] and [gRFC A96][]. It +does not change channel behavior, target resolution, OpenTelemetry View +behavior, or the deprecated C++ target filter API. [gRFC A96]: A96-retry-otel-stats.md -### C++ - -gRPC C++ will add a method `SetTargetAttributeFilter` to -`OpenTelemetryPluginBuilder` that accepts an -`absl::AnyInvocable`. - -```cpp -OpenTelemetryPluginBuilder& SetTargetAttributeFilter( - absl::AnyInvocable - target_attribute_filter); -``` - -The filter is stored in the plugin state and applied when creating -`OpenTelemetryClientFilter`. If no filter is registered or if the filter returns -`true`, the original target string is used. Otherwise, `"other"` is used. - - ### Java -gRPC Java will add a new method `targetAttributeFilter` to -`GrpcOpenTelemetry.Builder` that accepts a `Predicate`. To ensure compatibility with Android -API levels < 24 (where `Predicate` is not available), the filter is converted and stored internally -using a package-private interface. +gRPC Java adds a new method `targetAttributeFilter` to +`GrpcOpenTelemetry.Builder` that accepts a `Predicate`. To ensure +compatibility with Android API levels < 24, where `Predicate` is not available +without desugaring, the filter is converted and stored internally using a +package-private interface. The filter defaults to `null` when unset, meaning all targets are recorded as-is. ```java +@ExperimentalApi("https://github.com/grpc/grpc-java/issues/12595") +@IgnoreJRERequirement public Builder targetAttributeFilter(@Nullable Predicate filter) ``` When a filter is provided, `filter.test(target)` is called when the client interceptor is created -for a channel. If the predicate returns `true`, the original target string is used as +for a channel. If the predicate returns `true`, the original target string is used as `grpc.target`. If it returns `false`, the string `"other"` is used instead. The filter is applied when the client interceptor is created, meaning the @@ -95,9 +115,15 @@ on that channel. ## Rationale -The `targetAttributeFilter` controls how the `grpc.target` attribute is recorded in OpenTelemetry metrics. Targets accepted by the filter are recorded as-is; rejected targets are replaced with `"other"` to limit metric cardinality. +The `targetAttributeFilter` controls how the `grpc.target` attribute is recorded +in gRPC Java OpenTelemetry client metrics. Targets accepted by the filter are +recorded as-is; rejected targets are replaced with `"other"` to limit metric +cardinality. -This approach is already implemented in gRPC C++, and bringing it to gRPC Java ensures consistent metric semantics across languages, which is important for multi-language deployments. +This addresses the concrete Java cardinality problem reported in +[grpc/grpc-java#12322][]. This proposal only allows applications to keep the +original target or replace it with `"other"` for Java OpenTelemetry client +metrics; it is not a general API for rewriting target names across gRPC. **Alternative approach considered:** @@ -108,45 +134,16 @@ This approach is already implemented in gRPC C++, and bringing it to gRPC Java e * Simple and straightforward to implement. * Immediately addresses high-cardinality metrics. -* Aligns Java behavior with existing C++ implementation. +* Preserves existing behavior by default. +* Allows selective target retention, unlike an OpenTelemetry View that removes + `grpc.target` entirely. ## Implementation -### C++ - -The implementation adds `SetTargetAttributeFilter` to -`OpenTelemetryPluginBuilder`: - -```cpp -OpenTelemetryPluginBuilder& -OpenTelemetryPluginBuilder::SetTargetAttributeFilter( - absl::AnyInvocable - target_attribute_filter) { - target_attribute_filter_ = std::move(target_attribute_filter); - return *this; -} -``` - -The filter is stored in the plugin state and applied when creating -`OpenTelemetryClientFilter`: - -```cpp -absl::StatusOr OpenTelemetryClientFilter::Create( - const grpc_core::ChannelArgs& args, ChannelFilter::Args /*filter_args*/) { - std::string target = - args.GetOwnedString(GRPC_ARG_SERVER_URI).value_or(""); - - if (OTelPluginState().target_attribute_filter == nullptr || - OTelPluginState().target_attribute_filter(target)) { - return OpenTelemetryClientFilter(std::move(target)); - } - return OpenTelemetryClientFilter("other"); -} -``` - ### Java -The implementation adds `targetAttributeFilter` to `GrpcOpenTelemetry.Builder` and uses an internal +The Java implementation was merged in [grpc/grpc-java#12587][]. It adds +`targetAttributeFilter` to `GrpcOpenTelemetry.Builder` and uses an internal interface for storage. ```java @@ -158,12 +155,7 @@ public Builder targetAttributeFilter(@Nullable Predicate filter) { if (filter == null) { this.targetFilter = null; } else { - this.targetFilter = new TargetFilter() { - @Override - public boolean test(String target) { - return filter.test(target); - } - }; + this.targetFilter = filter::test; } return this; }