Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 66 additions & 1 deletion docs/develop/dotnet/platform/observability.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: observability
title: Observability - .NET SDK
title: Observability
sidebar_label: Observability
description: Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more.
toc_max_heading_level: 4
Expand Down Expand Up @@ -74,6 +74,71 @@ var runtime = new TemporalRuntime(new()
var client = await Temporalio.ConnectAsync(new("localhost:7233") { Runtime = runtime });
```

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`GlobalTags`](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html#Temporalio_Runtime_MetricsOptions_GlobalTags) on the [`Metrics` telemetry options](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html) to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker.

```csharp
using Temporalio.Client;
using Temporalio.Runtime;

var runtime = new TemporalRuntime(new()
{
Telemetry = new()
{
Metrics = new()
{
Prometheus = new("0.0.0.0:9000"),
GlobalTags = new Dictionary<string, string>
{
["team"] = "content-platform",
["service"] = "checkout",
["cost_center"] = "cc-1042",
["environment"] = "production",
},
},
},
});
var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime });
```

#### Choose a tag set

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Move shared tag policy out of per-SDK pages

The entire “Choose a tag set” block is language-neutral organizational policy and is duplicated across all six modified observability pages: .NET, Go, Java, Python, Ruby, and TypeScript. This turns SDK how-to pages into repeated best-practice guidance that can drift independently; move the shared policy to one language-neutral Best Practices page and keep only a link plus the SDK-specific configuration here.

AGENTS.md reference: AGENTS.md:L247-L253

Useful? React with 👍 / 👎.


Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag | Example | Question it answers |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service` | `checkout` | Which application emits these metrics? |
| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? |
| `environment` | `production` | Is this production traffic, or staging or test? |
| `region` | `us-east-2` | Where does the Worker fleet run? |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.
Comment on lines +125 to +127

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Lead the incident guidance with the reader's action

In each of the six copies, this incident passage spends three sentences describing an abstract gap before telling the reader what to do. Rewrite it as direct triage steps—group by team, check that team's recent deployments, and investigate a shared cause when multiple team values are affected—then state the rationale.

AGENTS.md reference: AGENTS.md:L266-L268

Useful? React with 👍 / 👎.


Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Setup Tracing {/* #tracing */}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📝 [vale] <Temporal.Headings> reported by reviewdog 🐶
'Setup Tracing ****************' should use sentence-style capitalization.


Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows.
Expand Down
66 changes: 65 additions & 1 deletion docs/develop/go/platform/observability.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: observability
title: Observability - Go SDK
title: Observability
sidebar_label: Observability
toc_max_heading_level: 4
tags:
Expand Down Expand Up @@ -53,6 +53,70 @@ The Go SDK provides metrics handlers for [Tally](https://pkg.go.dev/go.temporal.

For more information, see the [Go sample for metrics](https://github.com/temporalio/samples-go/tree/main/metrics).

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Call [`WithTags`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsHandler) on the metrics handler before you set it on the Client Options.
Every metric created from that handler carries the tags, from both the Client and the Worker.

```go
func main() {
// Create the base OTel metrics handler
metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{})

// Add global/static tags to all emitted metrics
globalTagsHandler := metricsHandler.WithTags(map[string]string{
"team": "content-platform",
"service": "checkout",
"cost_center": "cc-1042",
"environment": "production",
})

// Attach the tagged handler to client options
clientOptions := client.Options{
MetricsHandler: globalTagsHandler,
}

temporalClient, err := client.Dial(clientOptions)
}
```

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag | Example | Question it answers |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service` | `checkout` | Which application emits these metrics? |
| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? |
| `environment` | `production` | Is this production traffic, or staging or test? |
| `region` | `us-east-2` | Where does the Worker fleet run? |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

### Configure OpenTelemetry counters as monotonic {/* #opentelemetry-monotonic-counters */}

:::note
Expand Down
66 changes: 65 additions & 1 deletion docs/develop/java/platform/observability.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
id: observability
title: Observability - Java SDK
title: Observability
sidebar_label: Observability
description: Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes.
toc_max_heading_level: 4
Expand Down Expand Up @@ -56,6 +56,70 @@ The following example shows how to use `MicrometerClientStatsReporter` to define
For more details, see the [Java SDK Samples](https://github.com/temporalio/samples-java/tree/637c2e66fd2dab43d9f3f39e5fd9c55e4f3884f0/core/src/main/java/io/temporal/samples/metrics).
For details on configuring a Prometheus scrape endpoint with Micrometer, see the [Micrometer Prometheus Configuring](https://docs.micrometer.io/micrometer/reference/implementations/prometheus.html#_configuring) documentation.

### Attach global tags to metrics

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Pass the tags to `RootScopeBuilder.tags` when you build the metrics scope.
Every metric reported through that scope carries them, from both the Client and the Worker.

```java
//...
// Set up prometheus registry and stats reported
PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
StatsReporter reporter = new MicrometerClientStatsReporter(registry);

Map<String, String> globalTags = new HashMap<>();
globalTags.put("team", "content-platform");
globalTags.put("service", "checkout");
globalTags.put("cost_center", "cc-1042");
globalTags.put("environment", "production");

Scope scope = new RootScopeBuilder()
.tags(globalTags)
.reporter(reporter)
.reportEvery(com.uber.m3.util.Duration.ofSeconds(10));

WorkflowServiceStubsOptions stubOptions =
WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build();
//...
```

#### Choose a tag set

Tags are most useful when standardized across the organization, so that every Worker emits the same keys.
Decide on the set before teams adopt it.
These five suit most organizations:

| Tag | Example | Question it answers |
| ------------- | ------------------ | --------------------------------------------------------- |
| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service` | `checkout` | Which application emits these metrics? |
| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? |
| `environment` | `production` | Is this production traffic, or staging or test? |
| `region` | `us-east-2` | Where does the Worker fleet run? |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing {/* #tracing */}

Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows.
Expand Down
63 changes: 61 additions & 2 deletions docs/develop/python/platform/observability.mdx
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
---
id: observability
title: Observability - Python SDK
title: Observability
Comment thread
flippedcoder marked this conversation as resolved.
sidebar_label: Observability
description: Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes.
toc_max_heading_level: 2
toc_max_heading_level: 4
tags:
- Observability
- Workflows
Expand Down Expand Up @@ -32,6 +32,8 @@ For a complete list of metrics capable of being emitted, see the [SDK metrics re

Metrics in Python are configured globally; therefore, you should set a Prometheus endpoint before any other Temporal code.

### Set a Prometheus endpoint

The following example exposes a Prometheus endpoint on port `9000`.

```python
Expand All @@ -43,6 +45,63 @@ new_runtime = Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_ad
my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime)
```

### Attach global tags to metrics
Comment thread
flippedcoder marked this conversation as resolved.

SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`.
Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them.

Set [`global_tags`](https://python.temporal.io/temporalio.runtime.TelemetryConfig.html#global_tags) on `TelemetryConfig` to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker.

```python
from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig

new_runtime = Runtime(
telemetry=TelemetryConfig(
metrics=PrometheusConfig(bind_address="0.0.0.0:9000"),
global_tags={
"team": "content-platform",
"service": "checkout",
"cost_center": "cc-1042",
"environment": "production",
},
)
)
my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime)
```

#### Choose a tag set

Tags group metrics work best when standardized across the organization. Every Worker across your organization emits the same keys, so decide on the set before teams adopt it.
These five suit most organizations:

| Tag | Example | Question it answers |
| ------------- | ------------------ | ------------------------------------------------------ |
| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? |
| `service` | `checkout` | Which application emits these metrics? |
| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? |
| `environment` | `production` | Is this production traffic, or staging or test? |
| `region` | `us-east-2` | Where does the Worker fleet run? |

The built-in tags identify where a metric came from inside Temporal.
`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question.

That gap costs you time during an incident.
When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently.
Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team.

Grouping by `team` also tells you which case you are looking at:

- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics.
- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause.

The same grouping answers questions outside incidents.
A `cost_center` tag shows which budget owner drives Workflow and Activity volume.
SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill.

Keep tag values low cardinality.
Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores.
Ownership and deployment identifiers avoid this because they stay fixed for the life of the process.

## Set up tracing {/* #tracing */}

Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows.
Expand Down
Loading