diff --git a/docs/develop/dotnet/platform/observability.mdx b/docs/develop/dotnet/platform/observability.mdx index 961e7eba1e..050fdecac1 100644 --- a/docs/develop/dotnet/platform/observability.mdx +++ b/docs/develop/dotnet/platform/observability.mdx @@ -1,6 +1,6 @@ --- id: observability -title: Observability - .NET SDK +title: Observability sidebar_label: Observability description: Explore Temporal SDK observability features for Metrics, Tracing, Logging, and Visibility. Track Workflow Executions, set up Prometheus endpoints, customize metrics, configure tracing, and more. toc_max_heading_level: 4 @@ -74,6 +74,71 @@ var runtime = new TemporalRuntime(new() var client = await Temporalio.ConnectAsync(new("localhost:7233") { Runtime = runtime }); ``` +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Set [`GlobalTags`](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html#Temporalio_Runtime_MetricsOptions_GlobalTags) on the [`Metrics` telemetry options](https://dotnet.temporal.io/api/Temporalio.Runtime.MetricsOptions.html) to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker. + +```csharp +using Temporalio.Client; +using Temporalio.Runtime; + +var runtime = new TemporalRuntime(new() +{ + Telemetry = new() + { + Metrics = new() + { + Prometheus = new("0.0.0.0:9000"), + GlobalTags = new Dictionary + { + ["team"] = "content-platform", + ["service"] = "checkout", + ["cost_center"] = "cc-1042", + ["environment"] = "production", + }, + }, + }, +}); +var client = await TemporalClient.ConnectAsync(new("localhost:7233") { Runtime = runtime }); +``` + +#### Choose a tag set + +Tags are most useful when standardized across the organization, so that every Worker emits the same keys. +Decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | --------------------------------------------------------- | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ## Setup Tracing {/* #tracing */} Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows. diff --git a/docs/develop/go/platform/observability.mdx b/docs/develop/go/platform/observability.mdx index 0909e51a83..ecb6ff99f3 100644 --- a/docs/develop/go/platform/observability.mdx +++ b/docs/develop/go/platform/observability.mdx @@ -1,6 +1,6 @@ --- id: observability -title: Observability - Go SDK +title: Observability sidebar_label: Observability toc_max_heading_level: 4 tags: @@ -53,6 +53,70 @@ The Go SDK provides metrics handlers for [Tally](https://pkg.go.dev/go.temporal. For more information, see the [Go sample for metrics](https://github.com/temporalio/samples-go/tree/main/metrics). +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Call [`WithTags`](https://pkg.go.dev/go.temporal.io/sdk/client#MetricsHandler) on the metrics handler before you set it on the Client Options. +Every metric created from that handler carries the tags, from both the Client and the Worker. + +```go +func main() { + // Create the base OTel metrics handler + metricsHandler := temporalotel.NewMetricsHandler(temporalotel.MetricsHandlerOptions{}) + + // Add global/static tags to all emitted metrics + globalTagsHandler := metricsHandler.WithTags(map[string]string{ + "team": "content-platform", + "service": "checkout", + "cost_center": "cc-1042", + "environment": "production", + }) + + // Attach the tagged handler to client options + clientOptions := client.Options{ + MetricsHandler: globalTagsHandler, + } + + temporalClient, err := client.Dial(clientOptions) +} +``` + +#### Choose a tag set + +Tags are most useful when standardized across the organization, so that every Worker emits the same keys. +Decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | --------------------------------------------------------- | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ### Configure OpenTelemetry counters as monotonic {/* #opentelemetry-monotonic-counters */} :::note diff --git a/docs/develop/java/platform/observability.mdx b/docs/develop/java/platform/observability.mdx index d31fa92896..844bac0782 100644 --- a/docs/develop/java/platform/observability.mdx +++ b/docs/develop/java/platform/observability.mdx @@ -1,6 +1,6 @@ --- id: observability -title: Observability - Java SDK +title: Observability sidebar_label: Observability description: Explore the observability features of Temporal, including Metrics, Tracing, Logging, and Visibility. Emit Metrics with the Java SDK, set up Tracing, and use Search Attributes. toc_max_heading_level: 4 @@ -56,6 +56,70 @@ The following example shows how to use `MicrometerClientStatsReporter` to define For more details, see the [Java SDK Samples](https://github.com/temporalio/samples-java/tree/637c2e66fd2dab43d9f3f39e5fd9c55e4f3884f0/core/src/main/java/io/temporal/samples/metrics). For details on configuring a Prometheus scrape endpoint with Micrometer, see the [Micrometer Prometheus Configuring](https://docs.micrometer.io/micrometer/reference/implementations/prometheus.html#_configuring) documentation. +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Pass the tags to `RootScopeBuilder.tags` when you build the metrics scope. +Every metric reported through that scope carries them, from both the Client and the Worker. + +```java +//... +// Set up prometheus registry and stats reported + PrometheusMeterRegistry registry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); + StatsReporter reporter = new MicrometerClientStatsReporter(registry); + + Map globalTags = new HashMap<>(); + globalTags.put("team", "content-platform"); + globalTags.put("service", "checkout"); + globalTags.put("cost_center", "cc-1042"); + globalTags.put("environment", "production"); + + Scope scope = new RootScopeBuilder() + .tags(globalTags) + .reporter(reporter) + .reportEvery(com.uber.m3.util.Duration.ofSeconds(10)); + + WorkflowServiceStubsOptions stubOptions = + WorkflowServiceStubsOptions.newBuilder().setMetricsScope(scope).build(); +//... +``` + +#### Choose a tag set + +Tags are most useful when standardized across the organization, so that every Worker emits the same keys. +Decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | --------------------------------------------------------- | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ## Set up tracing {/* #tracing */} Tracing allows you to view the call graph of a Workflow along with its Activities, Nexus Operations, and any Child Workflows. diff --git a/docs/develop/python/platform/observability.mdx b/docs/develop/python/platform/observability.mdx index b382788b76..979cb745fc 100644 --- a/docs/develop/python/platform/observability.mdx +++ b/docs/develop/python/platform/observability.mdx @@ -1,9 +1,9 @@ --- id: observability -title: Observability - Python SDK +title: Observability sidebar_label: Observability description: Discover how to monitor your Temporal Application using metrics, tracing, logging, and visibility APIs. Emit metrics, set up tracing, log from Workflows, and use custom Search Attributes. -toc_max_heading_level: 2 +toc_max_heading_level: 4 tags: - Observability - Workflows @@ -32,6 +32,8 @@ For a complete list of metrics capable of being emitted, see the [SDK metrics re Metrics in Python are configured globally; therefore, you should set a Prometheus endpoint before any other Temporal code. +### Set a Prometheus endpoint + The following example exposes a Prometheus endpoint on port `9000`. ```python @@ -43,6 +45,63 @@ new_runtime = Runtime(telemetry=TelemetryConfig(metrics=PrometheusConfig(bind_ad my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime) ``` +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Set [`global_tags`](https://python.temporal.io/temporalio.runtime.TelemetryConfig.html#global_tags) on `TelemetryConfig` to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker. + +```python +from temporalio.runtime import Runtime, TelemetryConfig, PrometheusConfig + +new_runtime = Runtime( + telemetry=TelemetryConfig( + metrics=PrometheusConfig(bind_address="0.0.0.0:9000"), + global_tags={ + "team": "content-platform", + "service": "checkout", + "cost_center": "cc-1042", + "environment": "production", + }, + ) +) +my_client = await Client.connect("my.temporal.host:7233", runtime=new_runtime) +``` + +#### Choose a tag set + +Tags group metrics work best when standardized across the organization. Every Worker across your organization emits the same keys, so decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | ------------------------------------------------------ | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ## Set up tracing {/* #tracing */} Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows. diff --git a/docs/develop/ruby/platform/observability.mdx b/docs/develop/ruby/platform/observability.mdx index ac2999b198..6d8ce16928 100644 --- a/docs/develop/ruby/platform/observability.mdx +++ b/docs/develop/ruby/platform/observability.mdx @@ -53,6 +53,66 @@ Temporalio::Runtime.default = Temporalio::Runtime.new( Instead of Prometheus or OpenTelemetry, an instance of `Temporalio::Runtime::MetricBuffer` can be provided as a `buffer` argument to the `MetricsOptions`. `retrieve_updates` can then be periodically called on the buffer to get metric updates. +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Set [`global_tags`](https://ruby.temporal.io/Temporalio/Runtime/MetricsOptions.html#global_tags-instance_method) on the `MetricsOptions` to add the same key-value pairs to every metric the runtime emits, from both the Client and the Worker. + +```ruby +Temporalio::Runtime.default = Temporalio::Runtime.new( + telemetry: Temporalio::Runtime::TelemetryOptions.new( + metrics: Temporalio::Runtime::MetricsOptions.new( + opentelemetry: Temporalio::Runtime::OpenTelemetryMetricsOptions.new( + url: 'http://127.0.0.1:4317', + durations_as_seconds: true + ), + global_tags: { + 'team' => 'content-platform', + 'service' => 'checkout', + 'cost_center' => 'cc-1042', + 'environment' => 'production' + } + ) + ) +) +``` + +### Choose a tag set + +Tags are most useful when standardized across the organization, so that every Worker emits the same keys. +Decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | --------------------------------------------------------- | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ## Setup Tracing {/* #tracing */} Tracing enables observability into the sequence of calls across your application, including Workflows and Activities. diff --git a/docs/develop/typescript/platform/observability.mdx b/docs/develop/typescript/platform/observability.mdx index 0714b75882..2093346de6 100644 --- a/docs/develop/typescript/platform/observability.mdx +++ b/docs/develop/typescript/platform/observability.mdx @@ -3,7 +3,7 @@ id: observability title: Observability - TypeScript SDK sidebar_label: Observability description: Enhance the observability of your Temporal Application with metrics, tracing, logging, and visibility features. View Workflow state, set up OpenTelemetry, and customize logging for seamless monitoring and insights. -toc_max_heading_level: 3 +toc_max_heading_level: 4 tags: - Observability - Workflows @@ -47,6 +47,66 @@ telemetryOptions: { }, ``` +### Attach global tags to metrics + +SDK metrics arrive tagged with Temporal information such as `namespace` and `task_queue`. +Global tags add your organization's information next to them, so a dashboard can group Workers by the team, service, or environment that owns them. + +Set [`globalTags`](https://typescript.temporal.io/api/namespaces/worker#metricsexporterconfig) on the metrics telemetry options passed to [`Runtime.install`](https://typescript.temporal.io/api/classes/worker.Runtime/#install). + +```typescript +Runtime.install({ + telemetryOptions: { + metrics: { + prometheus: { bindAddress: '0.0.0.0:9464' }, + globalTags: { + team: 'content-platform', + service: 'checkout', + cost_center: 'cc-1042', + environment: 'production', + }, + }, + }, +}); +``` + +Global tags reach the Prometheus and OpenTelemetry exporters. +A buffered metrics exporter does not receive them. + +#### Choose a tag set + +Tags are most useful when standardized across the organization, so that every Worker emits the same keys. +Decide on the set before teams adopt it. +These five suit most organizations: + +| Tag | Example | Question it answers | +| ------------- | ------------------ | --------------------------------------------------------- | +| `team` | `content-platform` | Who owns the Workers behind this Namespace or Task Queue? | +| `service` | `checkout` | Which application emits these metrics? | +| `cost_center` | `cc-1042` | Which budget does this Worker fleet belong to? | +| `environment` | `production` | Is this production traffic, or staging or test? | +| `region` | `us-east-2` | Where does the Worker fleet run? | + +The built-in tags identify where a metric came from inside Temporal. +`namespace` and `task_queue` do not record which team runs the Workers behind them, so a dashboard grouped only by those tags cannot answer an ownership question. + +That gap costs you time during an incident. +When several Namespaces degrade at once, what you need first is the name of the team that owns the affected Workers, so you can ask whether they deployed recently. +Standardized tags put that name on the dashboard, which turns a broad question about the Temporal Service into a direct message to one team. + +Grouping by `team` also tells you which case you are looking at: + +- The affected Workers share one `team` value. Check that team's recent deploys first, because a deploy that restarts a Worker fleet causes a short disturbance in its metrics. +- The affected Workers span several `team` values. A single team's deploy no longer explains the pattern, so you can rule it out and look for a shared cause. + +The same grouping answers questions outside incidents. +A `cost_center` tag shows which budget owner drives Workflow and Activity volume. +SDK metrics count what your Workers and Clients do, which is not the same as the [Actions](/cloud/pricing#action) Temporal Cloud bills for, so use them to compare teams rather than to reconcile a bill. + +Keep tag values low cardinality. +Your metrics backend stores one series per distinct combination of tag values, so a value that changes per Workflow Execution, such as a Workflow Id or a customer identifier, multiplies what it stores. +Ownership and deployment identifiers avoid this because they stay fixed for the life of the process. + ## Set up tracing {/* #tracing */} Tracing allows you to view the call graph of a Workflow along with its Activities and any Child Workflows.