A121: RPC Delay Observability by mbissa · Pull Request #556 · grpc/proposal

mbissa · 2026-06-23T03:51:20Z

No description provided.

…eep simple

AgraVator · 2026-06-26T09:29:23Z

-*   **Pass-Through Container Policies**: Policies like `xds_cluster_manager`, `weighted_target`, and `rls` **do not prepend any prefix or wrap the delay type**. They simply bubble up the child's `grpc.delay_type` (e.g., `"connecting"`) directly as-is.
+*   **Pass-Through Container Policies**: Policies like `xds_cluster_manager`, `weighted_target`, and `rls` **do not prepend any prefix or wrap the delay type**. They bubble up the child's `grpc.delay_type` (e.g., `"connecting"`) as-is.
+
+Because only the priority policy contributes a prefix and it contributes exactly one (its active tier index), the resulting `grpc.delay_type` cardinality stays bounded; prefixes do not stack across nested pass-through containers. Implementations should keep the prefix a small bounded token (the tier index) so metric cardinality remains low; detailed per-container structure belongs in the `grpc.delay_reason` span event, not the metric label.


We are not gonna limit the nesting for priority policies right(as in px:py:pz.....:connecting ) ??

markdroth

The design looks good overall. I have one or two minor questions about design details, but they don't affect the overall shape of the design. Most of my comments are about the organization of the document -- the content is good but is not organized in a way that makes it easy to understand.

Please let me know if you have any questions. Thanks!

markdroth · 2026-07-13T23:05:09Z

@@ -0,0 +1,324 @@
+# A121: RPC Delay Observability


Please remove the # here, since this results in (a) a double underline in the formatted view and (b) adding the title to the table of contents, which isn't useful.

markdroth · 2026-07-14T16:23:19Z

+
+Existing gRPC core telemetry infrastructure, as defined in [gRPC A66 (OpenTelemetry Metrics)][A66] and [gRPC A72 (OpenTelemetry Tracing)][A72], tracks the overall end-to-end duration of RPC calls and attempts. However, these metrics and spans function as aggregate buckets that do not decompose latency, leaving delays inside the client channel invisible to operators.
+
+Before an RPC attempt can be sent over the network, the client channel must perform several operations, including resolving the target name, parsing service configurations, instantiating load balancing policies, and obtaining a connectivity picker. If any of these phases stall—such as during a slow DNS lookup, a Route Lookup Service (RLS) control-plane query, or a Cluster Discovery Service (CDS) metadata fetch—the RPC is delayed. To the application, this appears as high latency or a timeout. However, because current telemetry lacks visibility into these resolution and routing states, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay. This is particularly challenging for clients using the Route Lookup Service (RLS), where diagnosing RLS-related hangs or isolating whether delays stem from pending route lookups requires manual debugging.


A few comments here:

Parsing the service config is synchronous and does not impose any delay. Same thing for instantiating LB policies. I don't think we should mention either one here, since they're not relevant.

RLS is not actually a public API and therefore shouldn't be covered too much here. There should be only one place where we discuss it, which is where we talk about the changes in the RLS LB policy, and we should have a note on that section that says something like "(not a public API, not covered in any gRFC)".

Suggested change

Before an RPC attempt can be sent over the network, the client channel must perform several operations, including resolving the target name, parsing service configurations, instantiating load balancing policies, and obtaining a connectivity picker. If any of these phases stall—such as during a slow DNS lookup, a Route Lookup Service (RLS) control-plane query, or a Cluster Discovery Service (CDS) metadata fetch—the RPC is delayed. To the application, this appears as high latency or a timeout. However, because current telemetry lacks visibility into these resolution and routing states, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay. This is particularly challenging for clients using the Route Lookup Service (RLS), where diagnosing RLS-related hangs or isolating whether delays stem from pending route lookups requires manual debugging.

When a gRPC channel is first created, it is in IDLE state, meaning that it has not done name resolution or attempted to connect to any endpoints. When the application sends an RPC on the channel, the channel will leave IDLE state and attempt to connect. Any RPCs sent on the channel while it is attempting to connect will be queued until the channel becomes connected, which results in increased latency for these RPCs. However, because current telemetry lacks visibility into these connecting delays, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay.

markdroth · 2026-07-14T16:25:19Z

+
+Before an RPC attempt can be sent over the network, the client channel must perform several operations, including resolving the target name, parsing service configurations, instantiating load balancing policies, and obtaining a connectivity picker. If any of these phases stall—such as during a slow DNS lookup, a Route Lookup Service (RLS) control-plane query, or a Cluster Discovery Service (CDS) metadata fetch—the RPC is delayed. To the application, this appears as high latency or a timeout. However, because current telemetry lacks visibility into these resolution and routing states, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay. This is particularly challenging for clients using the Route Lookup Service (RLS), where diagnosing RLS-related hangs or isolating whether delays stem from pending route lookups requires manual debugging.
+
+While there are many potential sources of delay along the client-side pipeline (including interceptor execution, credential fetching, and filter evaluation), this proposal focuses specifically on introducing observability for the two most common bottlenecks: **name resolution** and **load balancing pick** delays. This proposal establishes a generic telemetry framework extensible to other client-side delays in the future.


Interceptor execution and filter evaluation are not relevant. We care only about things that do network I/O.

Credential fetching is a good example to cite here. Another example is retry delays.

markdroth · 2026-07-14T16:30:03Z

+{
+  "name": "Delay state transition",
+  "attributes": {
+    "grpc.delay_type": "string (the active delay type, e.g., 'connecting')",


Why do we need this field? The purpose of this event is to record the delay reason, but the delay type should be constant for the entire span, so it seems like we shouldn't need to record the delay type in each event.

markdroth · 2026-07-14T16:47:01Z

+| **Unit** | `s` (seconds) |
+| **Description** | EXPERIMENTAL. Time an RPC spent waiting at the call level before an attempt was initiated, such as waiting for name resolution. |
+| **Labels** | `grpc.target`, `grpc.delay_type` |
+| **Optional Labels** | `grpc.method` |


Why is the method label optional? I think it's a required label on all of the A66 metrics.

markdroth · 2026-07-14T18:26:50Z

+*   The core channel **MUST** invoke the tracer's end callback (`recordCallDelayEnd()` or `recordAttemptDelayEnd()`), which finalizes the plugin-side timing for the open segment. 
+*   This ensures the telemetry plugin can capture the elapsed duration up to the point of failure, close the trace span, and emit the partial duration metric, guaranteeing that bottlenecks preceding a failure remain observable.
+
+##### 4. Retries & Hedging


I think this section can just be removed. The structure it describes already exists and isn't something new in this design.

markdroth · 2026-07-14T18:28:56Z

+
+##### 3. Cancellation & Timeout
+If an active RPC call or attempt is cancelled (due to client cancellation, `DEADLINE_EXCEEDED` timeouts, or channel shutdown) while a delay is logically active:
+*   The core channel **MUST** invoke the tracer's end callback (`recordCallDelayEnd()` or `recordAttemptDelayEnd()`), which finalizes the plugin-side timing for the open segment. 


Do we actually need to explicitly close the span in this case? Note that when the RPC is cancelled or hits its deadline, we are already notifying the call tracer about that. We could just say that if there's an open delay when the call gets cancelled, the call tracer will automatically terminate it, updating both metrics and tracing accordingly.

markdroth · 2026-07-14T18:31:04Z

+The binding *mechanism* is asymmetric because each runtime's current telemetry API differs; only the six logical operations and their two scopes are common. This asymmetry is inherent to the existing APIs, not a difference in the delay design itself:
+*   **Go**: introduces a **new** call-scoped and attempt-scoped tracer API (gRPC-Go's V2 stats handler). A new API is required because the V1 `stats.Handler` is attempt-scoped only and cannot host a call-level hook. The broader V2 API is out of scope here beyond the delay hooks it must expose.
+*   **Java**: **reuses the existing tracer types** but adds new no-op-default methods to them — attempt hooks on `ClientStreamTracer`, call hooks on `ClientStreamTracer.Factory` — and reuses the existing name-resolution-delay plumbing (`ClientStreamTracer.NAME_RESOLUTION_DELAYED` and the `createPendingStream()` callback) for the `"resolving"` segment. The attempt-level `"connecting"` delay is recorded directly on the attempt-scoped `ClientStreamTracer` instance.
+*   **C++ (Core)**: **reuses the existing annotation framework**, adding only a new `DelayAnnotation` subtype that replaces the current free-form `"Delayed name resolution complete."` / `"Delayed LB pick complete."` string annotations.


I don't think we can do this via simple annotations. I think we'll instead add new methods to the call tracer API.

markdroth · 2026-07-14T18:36:42Z

+All delay metrics, tracing, and API hooks will be guarded by a feature flag:
+* **Go/Java Env Var**: `GRPC_EXPERIMENTAL_ENABLE_DELAY_OBSERVABILITY` (Default: `false`)
+* **C++ Core Experiment**: register `client_delay_observability` in `experiments.yaml`; gate via the generated `IsClientDelayObservabilityEnabled()` accessor (core uses generated per-experiment accessors, not string-keyed lookups).


I don't think we need any env var guard for this, since the new functionality is going to be configured locally instead of being triggered by remote input.

(Also, if we did need an env var guard, we'd use the same approach in C-core rather than using our experiment framework, so that interop tests have a single way to enable them across all languages.)

markdroth · 2026-07-14T18:37:57Z

+
+## Metric Registration and Recording
+
+Metric registration and recording will follow the established architectural patterns defined in [gRPC A66][A66] and [gRPC A94][A94]. The channel and attempts will delegate duration measurement directly to the registered telemetry plugin (e.g., OpenTelemetry) to avoid code duplication across the core stack. The specific plugin API registrations are omitted here for brevity but will conform to language-idiomatic OTel APIs (e.g., `RegisterFloat64Histo` in Go, `registerDoubleHistogram` in Java, and `RegisterDoubleHistogram` in C++).


I'm not sure what this is actually saying. Isn't this already implicit in the design described above?

I'm also not sure why this is referencing A94, since that doesn't seem relevant here.

mbissa added 6 commits April 23, 2026 16:01

draft lb delay metric

75e1539

delay metric rough

631f5f9

newer changes after review

8ca5bbc

go v2 change and cleanup

5cbe0c0

A121 gRFC title and filename renaming

d39ed0f

fixing title and filename

386b1f8

mbissa self-assigned this Jun 23, 2026

v1 feature parity for go stats handler and detailed rationale

a7b6979

mbissa assigned easwars, ejona86, markdroth and dfawley Jun 23, 2026

AgraVator mentioned this pull request Jun 23, 2026

core, opentelemetry: Implement Attempt-Level RPC Delay Observability (Proposal A121) grpc/grpc-java#12807

Open

AgraVator reviewed Jun 23, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

Comment thread A121-rpc-delay-observability.md

Comment thread A121-rpc-delay-observability.md Outdated

AgraVator reviewed Jun 23, 2026

View reviewed changes

Comment thread A121-rpc-delay-observability.md Outdated

mbissa requested review from dfawley, easwars, ejona86 and markdroth June 23, 2026 21:52

mbissa added 3 commits June 25, 2026 14:00

descope go v2 api verbose changes and cleanup api change section to k…

db81e29

…eep simple

add special section for attempt level delays handled by channel

6f6efbb

addressing review comment

5d8c3e1

mbissa removed their assignment Jun 25, 2026

mbissa added 2 commits June 25, 2026 17:50

fix AI slop

8265fe0

cleaning up some more

f6b5128

AgraVator reviewed Jun 26, 2026

View reviewed changes

markdroth reviewed Jul 14, 2026

View reviewed changes


		Existing gRPC core telemetry infrastructure, as defined in [gRPC A66 (OpenTelemetry Metrics)][A66] and [gRPC A72 (OpenTelemetry Tracing)][A72], tracks the overall end-to-end duration of RPC calls and attempts. However, these metrics and spans function as aggregate buckets that do not decompose latency, leaving delays inside the client channel invisible to operators.

		Before an RPC attempt can be sent over the network, the client channel must perform several operations, including resolving the target name, parsing service configurations, instantiating load balancing policies, and obtaining a connectivity picker. If any of these phases stall—such as during a slow DNS lookup, a Route Lookup Service (RLS) control-plane query, or a Cluster Discovery Service (CDS) metadata fetch—the RPC is delayed. To the application, this appears as high latency or a timeout. However, because current telemetry lacks visibility into these resolution and routing states, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay. This is particularly challenging for clients using the Route Lookup Service (RLS), where diagnosing RLS-related hangs or isolating whether delays stem from pending route lookups requires manual debugging.


		Before an RPC attempt can be sent over the network, the client channel must perform several operations, including resolving the target name, parsing service configurations, instantiating load balancing policies, and obtaining a connectivity picker. If any of these phases stall—such as during a slow DNS lookup, a Route Lookup Service (RLS) control-plane query, or a Cluster Discovery Service (CDS) metadata fetch—the RPC is delayed. To the application, this appears as high latency or a timeout. However, because current telemetry lacks visibility into these resolution and routing states, developers cannot distinguish between a slow network, a slow backend, or a channel initialization delay. This is particularly challenging for clients using the Route Lookup Service (RLS), where diagnosing RLS-related hangs or isolating whether delays stem from pending route lookups requires manual debugging.

		While there are many potential sources of delay along the client-side pipeline (including interceptor execution, credential fetching, and filter evaluation), this proposal focuses specifically on introducing observability for the two most common bottlenecks: name resolution and load balancing pick delays. This proposal establishes a generic telemetry framework extensible to other client-side delays in the future.


		## Metric Registration and Recording

		Metric registration and recording will follow the established architectural patterns defined in [gRPC A66][A66] and [gRPC A94][A94]. The channel and attempts will delegate duration measurement directly to the registered telemetry plugin (e.g., OpenTelemetry) to avoid code duplication across the core stack. The specific plugin API registrations are omitted here for brevity but will conform to language-idiomatic OTel APIs (e.g., `RegisterFloat64Histo` in Go, `registerDoubleHistogram` in Java, and `RegisterDoubleHistogram` in C++).

Uh oh!

Conversation

mbissa commented Jun 23, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

markdroth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants