OBSDOCS-3855: Document high availability for TempoStack

_topic_maps/_topic_map.yml

@@ -26,6 +26,8 @@ Distros: openshift-distributed-tracing
 Topics:
 - Name: Configuring distributed tracing
   File: distr-tracing-tempo-configuring
+- Name: Configuring high availability
+  File: distr-tracing-tempo-high-availability
 ---
 Name: Troubleshooting distributed tracing
 Dir: troubleshooting

configuring/distr-tracing-tempo-high-availability.adoc

@@ -0,0 +1,26 @@
+:_mod-docs-content-type: ASSEMBLY
+include::_attributes/common-attributes.adoc[]
+[id="distr-tracing-tempo-high-availability"]
+= Configuring high availability for {dt}
+:context: distr-tracing-tempo-high-availability
+
+toc::[]
+
+[role="_abstract"]
+You can configure replication, component replicas, pod placement, and zone awareness so that a TempoStack instance keeps ingesting and querying traces when individual pods, nodes, or availability zones become unavailable.
+
+include::modules/distr-tracing-tempo-about-high-availability.adoc[leveloffset=+1]
+
+include::modules/distr-tracing-tempo-ha-replication-factor.adoc[leveloffset=+2]
+
+include::modules/distr-tracing-tempo-ha-component-replicas.adoc[leveloffset=+2]
+
+include::modules/distr-tracing-tempo-ha-pod-placement.adoc[leveloffset=+2]
+
+include::modules/distr-tracing-tempo-ha-pod-disruption-budgets.adoc[leveloffset=+2]
+
+include::modules/distr-tracing-tempo-about-zone-aware-replication.adoc[leveloffset=+1]
+
+include::modules/distr-tracing-tempo-config-zone-aware-replication.adoc[leveloffset=+2]
+
+include::modules/distr-tracing-tempo-zone-fail-recovery.adoc[leveloffset=+2]

modules/distr-tracing-tempo-about-high-availability.adoc

@@ -0,0 +1,34 @@
+// Module included in the following assemblies:
+//
+// * configuring/distr-tracing-tempo-high-availability.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="distr-tracing-tempo-about-high-availability_{context}"]
+= High availability for TempoStack
+
+[role="_abstract"]
+Configure high availability so that a TempoStack instance keeps ingesting and querying traces when individual pods, nodes, or availability zones become unavailable.
+
+A TempoStack instance is a set of independently scalable microservices: distributor, ingester, compactor, querier, and query front end, plus the optional gateway and metrics generator. The {TempoOperator} combines the following mechanisms to keep these components available:
+
+Replication factor::
+  The `spec.replicationFactor` field defines how many ingesters must acknowledge a span before the distributor accepts it. A replication factor greater than 1 means that a span survives the loss of an ingester.
+
+Component replicas::
+  The `spec.size` field defaults every component to at least 2 replicas on all sizes except `1x.demo`, so that no component is a single point of failure.
+
+Pod placement::
+  The Operator applies pod anti-affinity rules so that the scheduler spreads the pods of a component across different nodes and failure domains.
+
+Pod disruption budgets::
+  The Operator creates a `PodDisruptionBudget` object for each component, so that voluntary disruptions, such as node drains during a cluster update, cannot take down more than 1 replica of a component at a time.
+
+Zone-aware replication::
+  The optional `spec.replicationZones` field spreads the pods of every component across availability zones and configures the ingester ring to replicate spans across those zones, so that the instance survives the loss of an entire zone.
+
+[NOTE]
+====
+These mechanisms apply to the `TempoStack` custom resource (CR) only. A `TempoMonolithic` instance runs all components in a single pod and is therefore not highly available.
+====
+
+The `1x.demo` size is for demonstration and development purposes only: it runs a single replica of every component with a replication factor of 1. For production deployments, use the `1x.pico` size or larger.

modules/distr-tracing-tempo-about-zone-aware-replication.adoc

@@ -0,0 +1,40 @@
+// Module included in the following assemblies:
+//
+// * configuring/distr-tracing-tempo-high-availability.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="distr-tracing-tempo-about-zone-aware-replication_{context}"]
+= Zone-aware data replication
+
+[role="_abstract"]
+Enable zone-aware data replication to protect against trace loss during an availability zone failure.
+
+Availability zones are isolated areas within the data center of a cloud provider, aimed at enhancing redundancy and fault tolerance. Without zone awareness, the scheduler can place all the ingester replicas that hold the copies of a span in the same zone, and a zone failure then loses every copy.
+
+The {TempoOperator} offers support for zone-aware data replication through pod topology spread constraints. When you configure the `spec.replicationZones` field, the Operator does the following:
+
+* Adds 1 topology spread constraint per configured zone to every component. Each constraint uses the `DoNotSchedule` policy, so the scheduler distributes the pods of a component evenly across the topology domains rather than only preferring to do so.
+* Configures the ingester ring with zone awareness, so that the ingesters that hold the copies of a span are in different zones.
+
+Because a pod cannot read the topology labels of the node that it runs on, the Operator determines the availability zone as follows:
+
+. The Operator labels the zone-aware pods with `tempo.grafana.com/zone-aware` and annotates them with the configured topology keys.
+. The `tempostack-zoneaware-pod` controller watches those pods, looks up the node that each pod was scheduled on, and patches the values of the configured topology labels onto the pod as the `tempo.grafana.com/availability-zone` annotation.
+. An `az-annotation-check` init container blocks the start of the pod until that annotation is populated. The annotation is exposed to the init container as a file through a downward API volume.
+. The annotation is passed to the Tempo container as the `INSTANCE_AVAILABILITY_ZONE` environment variable, which Tempo uses to register itself in the ring with its availability zone.
+
+The gateway is spread across the zones as well, but it does not join the hash ring, so it gets neither the init container nor the environment variable.
+
+[IMPORTANT]
+====
+The number of available zones must be greater than or equal to the value of the `spec.replicationFactor` field. The Operator does not validate this, because it cannot know how many distinct values of the topology key exist in the cluster. With fewer zones than the replication factor, the ingesters cannot satisfy the replication factor and writes fail.
+
+Each zone must host an equal number of nodes that are eligible to run the TempoStack pods. Otherwise, the `DoNotSchedule` policy leaves pods in the `Pending` state.
+====
+
+[NOTE]
+====
+* Zone-aware replication applies to the `TempoStack` custom resource (CR) only. The `TempoMonolithic` CR is unaffected.
+* Each zone must define a distinct topology key. The Operator rejects a `TempoStack` CR that defines the same topology key more than once.
+* Removing the `spec.replicationZones` field from an existing `TempoStack` CR removes the topology spread constraints and the init container again.
+====

modules/distr-tracing-tempo-config-size.adoc

@@ -57,59 +57,129 @@ The `TempoStack` instance size format is `1x.<size>`, where `1x` is the only sup
 |2
 |2
 
+|Highly available
+|No
+|Yes
+|Yes
+|Yes
+|Yes
+
 |Total CPU requests
 |None
-|3.25 vCPUs
-|4.6 vCPUs
-|7.9 vCPUs
-|24 vCPUs
+|5.5 vCPUs
+|6.2 vCPUs
+|14.9 vCPUs
+|50.4 vCPUs
 
 |Total CPU requests if using the gateway
 |None
-|3.4 vCPUs
-|5.1 vCPUs
-|8.8 vCPUs
-|28.1 vCPUs
+|5.8 vCPUs
+|7.2 vCPUs
+|16.7 vCPUs
+|58.8 vCPUs
 
 |Total CPU requests if using the Jaeger UI
 |None
-|3.4 vCPUs
-|5.1 vCPUs
-|8.8 vCPUs
-|28.1 vCPUs
+|5.8 vCPUs
+|7.2 vCPUs
+|16.7 vCPUs
+|58.8 vCPUs
 
 |Total memory requests
 |None
-|8.8 Gi
-|22.1 Gi
-|30.1 Gi
-|47.1 Gi
+|11.9 Gi
+|28.2 Gi
+|43.4 Gi
+|84.5 Gi
 
 |Total memory requests if using the gateway
 |None
-|8.9 Gi
-|22.3 Gi
-|30.4 Gi
-|47.4 Gi
+|12.2 Gi
+|28.6 Gi
+|43.9 Gi
+|85.1 Gi
 
 |Total memory requests if using the Jaeger UI
 |None
-|8.9 Gi
-|22.3 Gi
-|30.4 Gi
-|47.4 Gi
+|12.2 Gi
+|28.6 Gi
+|43.9 Gi
+|85.1 Gi
 |===
 
-For production deployments, use `1x.pico` or larger sizes that provide high availability with a replication factor of 2.
+For production deployments, use `1x.pico` or larger sizes, which provide high availability with a replication factor of 2 and at least 2 replicas of every component.
 
 [NOTE]
 ====
-The `1x.demo` size is for demonstration and development purposes only. It has no resource constraints and a replication factor of 1, so it does not provide high availability.
+The `1x.demo` size is for demonstration and development purposes only. It has no resource constraints, a replication factor of 1, and a single replica of every component; therefore, it does not provide high availability.
 ====
 
-The Operator automatically configures the resource requests for all TempoStack components and sets the appropriate replication factor. You do not need to manually configure individual component resources. The base resource totals include the following components: distributor, ingester, compactor, querier, and query front end. The gateway component is optional but required if you need multitenant authentication. When enabled, it also deploys the gateway Open Policy Agent (OPA) sidecar. The Jaeger UI component is optional and provides the Jaeger query interface for trace visualization. When enabled, it also deploys the OAuth proxy for authentication.
+The Operator automatically configures the resource requests for all TempoStack components, and sets the appropriate replication factor and per-component replica counts. You do not need to manually configure individual component resources. The base resource totals include the following components: distributor, ingester, compactor, querier, and query front end. The gateway component is optional but required if you need multitenant authentication. When enabled, it also deploys the gateway Open Policy Agent (OPA) sidecar. The Jaeger UI component is optional and provides the Jaeger query interface for trace visualization. When enabled, it also deploys the OAuth proxy for authentication. The metrics generator is optional and is not included in the totals.
+
+Each deployment size other than `1x.demo` runs at least 2 replicas of every component, so that the deployment is highly available, and scales the throughput-bound components at the larger sizes. The following table shows the default replica count for each component at each deployment size:
+
+[cols="1h,5*",options="header"]
+|===
+|Component
+|`1x.demo`
+|`1x.pico`
+|`1x.extra-small`
+|`1x.small`
+|`1x.medium`
+
+|Distributor
+|1
+|2
+|2
+|3
+|4
+
+|Ingester
+|1
+|2
+|2
+|2
+|2
+
+|Compactor
+|1
+|2
+|2
+|2
+|3
+
+|Querier
+|1
+|2
+|2
+|3
+|5
+
+|Query front end
+|1
+|2
+|2
+|2
+|2
+
+|Gateway
+|1
+|2
+|2
+|2
+|2
+
+|Metrics generator
+|1
+|2
+|2
+|3
+|3
+|===
+
+To override the default replica count of a component, set `spec.template.<component>.replicas`. An explicit replica count always takes precedence over the size profile.
 
-The following table shows the CPU and memory resource requests for each TempoStack component at each deployment size:
+The following table shows the CPU and memory resource requests for each replica of each TempoStack component at each deployment size:
 
 [cols="2h,2*,2*,2*,2*",options="header"]
 |===
@@ -223,9 +293,10 @@ The following table shows the CPU and memory resource requests for each TempoSta
 [NOTE]
 ====
 * The resource values listed are requests only. The Operator does not apply resource limits.
-* The replication factor determines the number of ingester replicas. For sizes with a replication factor of 2, multiply the ingester resources by 2 to calculate the total ingester resource consumption.
-* The Operator automatically deploys the gateway OPA sidecar when the gateway is enabled.
-* The Operator automatically deploys the OAuth proxy when the Jaeger UI is enabled.
+* The values are per replica. To calculate the total resource consumption of a component, multiply its values by the replica count of that component in the deployment size.
+* The replication factor determines the number of ingester replicas. The ingester runs `floor(replicationFactor / 2) + 1` replicas, which is 2 replicas for the sizes with a replication factor of 2.
+* The Operator automatically deploys the gateway OPA sidecar when the gateway is enabled. The gateway OPA sidecar runs in the gateway pod, so it scales with the gateway replica count.
+* The Operator automatically deploys the OAuth proxy when the Jaeger UI is enabled. The Jaeger UI and the OAuth proxy run in the query front end pod, so they scale with the query front end replica count.
 ====
 
 When determining resources for a component, the Operator uses the following priority order:
@@ -271,7 +342,7 @@ spec:
 ----
 where:
 
-`example`:: In this example, the following settings apply: The ingester component uses the explicitly defined resources, 4 CPU and 20 Gi memory, instead of the `1x.small` defaults. The querier component uses four replicas with resources from the `1x.small` profile. All the other components, such as the distributor, compactor, and query front end, use the `1x.small` default resources.
+`example`:: In this example, the following settings apply: The ingester component uses the explicitly defined resources, 4 CPU and 20 Gi memory, instead of the `1x.small` defaults. The querier component uses 4 replicas instead of the 3 replicas of the `1x.small` profile, with resources from the `1x.small` profile. All the other components, such as the distributor, compactor, and query front end, use the `1x.small` default resources and replica counts.
 `size`:: Specifies the base size profile for all components.
 `resources`:: Overrides the ingester resources. The ingester component uses these explicit values instead of the `1x.small` defaults.
-`replicas`:: Specifies the number of querier replicas. The replica counts are independent of the size profile.
+`replicas`:: Overrides the number of querier replicas. This value takes precedence over the replica count of the size profile.

modules/distr-tracing-tempo-config-zone-aware-replication.adoc

@@ -0,0 +1,80 @@
+// Module included in the following assemblies:
+//
+// * configuring/distr-tracing-tempo-high-availability.adoc
+
+:_mod-docs-content-type: PROCEDURE
+[id="distr-tracing-tempo-config-zone-aware-replication_{context}"]
+= Configuring zone-aware replication
+
+[role="_abstract"]
+Configure the `spec.replicationZones` field to spread the TempoStack pods across availability zones and to replicate spans across those zones.
+
+.Prerequisites
+
+* You have installed the {TempoOperator}.
+* The nodes of your cluster are labeled with the topology key that you intend to use, such as `topology.kubernetes.io/zone`.
+* The number of distinct availability zones is greater than or equal to the replication factor that you intend to configure.
+
+.Procedure
+
+. Configure the `spec.replicationZones` field in your `TempoStack` custom resource (CR).
++
+This example `TempoStack` CR has zone-aware replication enabled:
++
+[source,yaml]
+----
+apiVersion: tempo.grafana.com/v1alpha1
+kind: TempoStack
+metadata:
+  name: example
+  namespace: tracing-system
+spec:
+  size: 1x.small
+  replicationFactor: 3
+  replicationZones:
+  - maxSkew: 1
+    topologyKey: topology.kubernetes.io/zone
+  storage:
+    secret:
+      name: object-storage-secret
+      type: s3
+  template:
+    ingester:
+      replicas: 3
+----
+where:
+
+`spec.replicationFactor`:: Specifies how many ingesters must acknowledge a span. This value must be less than or equal to the number of available zones.
+`spec.replicationZones.maxSkew`:: Specifies the maximum difference in the number of pods of a component between any 2 topology domains. The default is 1, and you cannot specify a value lower than 1.
+`spec.replicationZones.topologyKey`:: Specifies a zone in the form of a topology key that corresponds to a node label.
+`spec.template.ingester.replicas`:: Specifies the number of ingester replicas. To place an ingester in each zone, set this value to the number of zones.
+
+. Apply the CR by running the following command:
++
+[source,terminal]
+----
+$ oc apply -f <tempostack_cr_file_name>.yaml -n tracing-system
+----
+
+.Verification
+
+. Verify that the pods of a component are distributed across the zones by running the following command:
++
+[source,terminal]
+----
+$ oc get pods -n tracing-system \
+    -l app.kubernetes.io/instance=example,app.kubernetes.io/component=ingester \
+    -o custom-columns='NAME:.metadata.name,ZONE:.metadata.annotations.tempo\.grafana\.com/availability-zone'
+----
++
+Example output:
++
+[source,terminal]
+----
+NAME                          ZONE
+tempo-example-ingester-0      us-east-1a
+tempo-example-ingester-1      us-east-1b
+tempo-example-ingester-2      us-east-1c
+----
+
+. If a pod remains in the `Init` state, verify that the `tempo.grafana.com/availability-zone` annotation is populated. An empty annotation means that the node that the pod was scheduled on is missing the configured topology label.