diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index b81f1cecca5..b57d6b2a8b0 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -26,6 +26,8 @@ Distros: openshift-distributed-tracing Topics: - Name: Configuring distributed tracing File: distr-tracing-tempo-configuring +- Name: Configuring high availability + File: distr-tracing-tempo-high-availability --- Name: Troubleshooting distributed tracing Dir: troubleshooting diff --git a/configuring/distr-tracing-tempo-high-availability.adoc b/configuring/distr-tracing-tempo-high-availability.adoc new file mode 100644 index 00000000000..b31a1f360cc --- /dev/null +++ b/configuring/distr-tracing-tempo-high-availability.adoc @@ -0,0 +1,26 @@ +:_mod-docs-content-type: ASSEMBLY +include::_attributes/common-attributes.adoc[] +[id="distr-tracing-tempo-high-availability"] += Configuring high availability for {dt} +:context: distr-tracing-tempo-high-availability + +toc::[] + +[role="_abstract"] +You can configure replication, component replicas, pod placement, and zone awareness so that a TempoStack instance keeps ingesting and querying traces when individual pods, nodes, or availability zones become unavailable. + +include::modules/distr-tracing-tempo-about-high-availability.adoc[leveloffset=+1] + +include::modules/distr-tracing-tempo-ha-replication-factor.adoc[leveloffset=+2] + +include::modules/distr-tracing-tempo-ha-component-replicas.adoc[leveloffset=+2] + +include::modules/distr-tracing-tempo-ha-pod-placement.adoc[leveloffset=+2] + +include::modules/distr-tracing-tempo-ha-pod-disruption-budgets.adoc[leveloffset=+2] + +include::modules/distr-tracing-tempo-about-zone-aware-replication.adoc[leveloffset=+1] + +include::modules/distr-tracing-tempo-config-zone-aware-replication.adoc[leveloffset=+2] + +include::modules/distr-tracing-tempo-zone-fail-recovery.adoc[leveloffset=+2] diff --git a/modules/distr-tracing-tempo-about-high-availability.adoc b/modules/distr-tracing-tempo-about-high-availability.adoc new file mode 100644 index 00000000000..96d448d56f8 --- /dev/null +++ b/modules/distr-tracing-tempo-about-high-availability.adoc @@ -0,0 +1,34 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: CONCEPT +[id="distr-tracing-tempo-about-high-availability_{context}"] += High availability for TempoStack + +[role="_abstract"] +Configure high availability so that a TempoStack instance keeps ingesting and querying traces when individual pods, nodes, or availability zones become unavailable. + +A TempoStack instance is a set of independently scalable microservices: distributor, ingester, compactor, querier, and query front end, plus the optional gateway and metrics generator. The {TempoOperator} combines the following mechanisms to keep these components available: + +Replication factor:: + The `spec.replicationFactor` field defines how many ingesters must acknowledge a span before the distributor accepts it. A replication factor greater than 1 means that a span survives the loss of an ingester. + +Component replicas:: + The `spec.size` field defaults every component to at least 2 replicas on all sizes except `1x.demo`, so that no component is a single point of failure. + +Pod placement:: + The Operator applies pod anti-affinity rules so that the scheduler spreads the pods of a component across different nodes and failure domains. + +Pod disruption budgets:: + The Operator creates a `PodDisruptionBudget` object for each component, so that voluntary disruptions, such as node drains during a cluster update, cannot take down more than 1 replica of a component at a time. + +Zone-aware replication:: + The optional `spec.replicationZones` field spreads the pods of every component across availability zones and configures the ingester ring to replicate spans across those zones, so that the instance survives the loss of an entire zone. + +[NOTE] +==== +These mechanisms apply to the `TempoStack` custom resource (CR) only. A `TempoMonolithic` instance runs all components in a single pod and is therefore not highly available. +==== + +The `1x.demo` size is for demonstration and development purposes only: it runs a single replica of every component with a replication factor of 1. For production deployments, use the `1x.pico` size or larger. diff --git a/modules/distr-tracing-tempo-about-zone-aware-replication.adoc b/modules/distr-tracing-tempo-about-zone-aware-replication.adoc new file mode 100644 index 00000000000..39755b06d3a --- /dev/null +++ b/modules/distr-tracing-tempo-about-zone-aware-replication.adoc @@ -0,0 +1,40 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: CONCEPT +[id="distr-tracing-tempo-about-zone-aware-replication_{context}"] += Zone-aware data replication + +[role="_abstract"] +Enable zone-aware data replication to protect against trace loss during an availability zone failure. + +Availability zones are isolated areas within the data center of a cloud provider, aimed at enhancing redundancy and fault tolerance. Without zone awareness, the scheduler can place all the ingester replicas that hold the copies of a span in the same zone, and a zone failure then loses every copy. + +The {TempoOperator} offers support for zone-aware data replication through pod topology spread constraints. When you configure the `spec.replicationZones` field, the Operator does the following: + +* Adds 1 topology spread constraint per configured zone to every component. Each constraint uses the `DoNotSchedule` policy, so the scheduler distributes the pods of a component evenly across the topology domains rather than only preferring to do so. +* Configures the ingester ring with zone awareness, so that the ingesters that hold the copies of a span are in different zones. + +Because a pod cannot read the topology labels of the node that it runs on, the Operator determines the availability zone as follows: + +. The Operator labels the zone-aware pods with `tempo.grafana.com/zone-aware` and annotates them with the configured topology keys. +. The `tempostack-zoneaware-pod` controller watches those pods, looks up the node that each pod was scheduled on, and patches the values of the configured topology labels onto the pod as the `tempo.grafana.com/availability-zone` annotation. +. An `az-annotation-check` init container blocks the start of the pod until that annotation is populated. The annotation is exposed to the init container as a file through a downward API volume. +. The annotation is passed to the Tempo container as the `INSTANCE_AVAILABILITY_ZONE` environment variable, which Tempo uses to register itself in the ring with its availability zone. + +The gateway is spread across the zones as well, but it does not join the hash ring, so it gets neither the init container nor the environment variable. + +[IMPORTANT] +==== +The number of available zones must be greater than or equal to the value of the `spec.replicationFactor` field. The Operator does not validate this, because it cannot know how many distinct values of the topology key exist in the cluster. With fewer zones than the replication factor, the ingesters cannot satisfy the replication factor and writes fail. + +Each zone must host an equal number of nodes that are eligible to run the TempoStack pods. Otherwise, the `DoNotSchedule` policy leaves pods in the `Pending` state. +==== + +[NOTE] +==== +* Zone-aware replication applies to the `TempoStack` custom resource (CR) only. The `TempoMonolithic` CR is unaffected. +* Each zone must define a distinct topology key. The Operator rejects a `TempoStack` CR that defines the same topology key more than once. +* Removing the `spec.replicationZones` field from an existing `TempoStack` CR removes the topology spread constraints and the init container again. +==== diff --git a/modules/distr-tracing-tempo-config-size.adoc b/modules/distr-tracing-tempo-config-size.adoc index d54a968d2de..fa709b08ccc 100644 --- a/modules/distr-tracing-tempo-config-size.adoc +++ b/modules/distr-tracing-tempo-config-size.adoc @@ -57,59 +57,129 @@ The `TempoStack` instance size format is `1x.`, where `1x` is the only sup |2 |2 +|Highly available +|No +|Yes +|Yes +|Yes +|Yes + |Total CPU requests |None -|3.25 vCPUs -|4.6 vCPUs -|7.9 vCPUs -|24 vCPUs +|5.5 vCPUs +|6.2 vCPUs +|14.9 vCPUs +|50.4 vCPUs |Total CPU requests if using the gateway |None -|3.4 vCPUs -|5.1 vCPUs -|8.8 vCPUs -|28.1 vCPUs +|5.8 vCPUs +|7.2 vCPUs +|16.7 vCPUs +|58.8 vCPUs |Total CPU requests if using the Jaeger UI |None -|3.4 vCPUs -|5.1 vCPUs -|8.8 vCPUs -|28.1 vCPUs +|5.8 vCPUs +|7.2 vCPUs +|16.7 vCPUs +|58.8 vCPUs |Total memory requests |None -|8.8 Gi -|22.1 Gi -|30.1 Gi -|47.1 Gi +|11.9 Gi +|28.2 Gi +|43.4 Gi +|84.5 Gi |Total memory requests if using the gateway |None -|8.9 Gi -|22.3 Gi -|30.4 Gi -|47.4 Gi +|12.2 Gi +|28.6 Gi +|43.9 Gi +|85.1 Gi |Total memory requests if using the Jaeger UI |None -|8.9 Gi -|22.3 Gi -|30.4 Gi -|47.4 Gi +|12.2 Gi +|28.6 Gi +|43.9 Gi +|85.1 Gi |=== -For production deployments, use `1x.pico` or larger sizes that provide high availability with a replication factor of 2. +For production deployments, use `1x.pico` or larger sizes, which provide high availability with a replication factor of 2 and at least 2 replicas of every component. [NOTE] ==== -The `1x.demo` size is for demonstration and development purposes only. It has no resource constraints and a replication factor of 1, so it does not provide high availability. +The `1x.demo` size is for demonstration and development purposes only. It has no resource constraints, a replication factor of 1, and a single replica of every component; therefore, it does not provide high availability. ==== -The Operator automatically configures the resource requests for all TempoStack components and sets the appropriate replication factor. You do not need to manually configure individual component resources. The base resource totals include the following components: distributor, ingester, compactor, querier, and query front end. The gateway component is optional but required if you need multitenant authentication. When enabled, it also deploys the gateway Open Policy Agent (OPA) sidecar. The Jaeger UI component is optional and provides the Jaeger query interface for trace visualization. When enabled, it also deploys the OAuth proxy for authentication. +The Operator automatically configures the resource requests for all TempoStack components, and sets the appropriate replication factor and per-component replica counts. You do not need to manually configure individual component resources. The base resource totals include the following components: distributor, ingester, compactor, querier, and query front end. The gateway component is optional but required if you need multitenant authentication. When enabled, it also deploys the gateway Open Policy Agent (OPA) sidecar. The Jaeger UI component is optional and provides the Jaeger query interface for trace visualization. When enabled, it also deploys the OAuth proxy for authentication. The metrics generator is optional and is not included in the totals. + +Each deployment size other than `1x.demo` runs at least 2 replicas of every component, so that the deployment is highly available, and scales the throughput-bound components at the larger sizes. The following table shows the default replica count for each component at each deployment size: + +[cols="1h,5*",options="header"] +|=== +|Component +|`1x.demo` +|`1x.pico` +|`1x.extra-small` +|`1x.small` +|`1x.medium` + +|Distributor +|1 +|2 +|2 +|3 +|4 + +|Ingester +|1 +|2 +|2 +|2 +|2 + +|Compactor +|1 +|2 +|2 +|2 +|3 + +|Querier +|1 +|2 +|2 +|3 +|5 + +|Query front end +|1 +|2 +|2 +|2 +|2 + +|Gateway +|1 +|2 +|2 +|2 +|2 + +|Metrics generator +|1 +|2 +|2 +|3 +|3 +|=== + +To override the default replica count of a component, set `spec.template..replicas`. An explicit replica count always takes precedence over the size profile. -The following table shows the CPU and memory resource requests for each TempoStack component at each deployment size: +The following table shows the CPU and memory resource requests for each replica of each TempoStack component at each deployment size: [cols="2h,2*,2*,2*,2*",options="header"] |=== @@ -223,9 +293,10 @@ The following table shows the CPU and memory resource requests for each TempoSta [NOTE] ==== * The resource values listed are requests only. The Operator does not apply resource limits. -* The replication factor determines the number of ingester replicas. For sizes with a replication factor of 2, multiply the ingester resources by 2 to calculate the total ingester resource consumption. -* The Operator automatically deploys the gateway OPA sidecar when the gateway is enabled. -* The Operator automatically deploys the OAuth proxy when the Jaeger UI is enabled. +* The values are per replica. To calculate the total resource consumption of a component, multiply its values by the replica count of that component in the deployment size. +* The replication factor determines the number of ingester replicas. The ingester runs `floor(replicationFactor / 2) + 1` replicas, which is 2 replicas for the sizes with a replication factor of 2. +* The Operator automatically deploys the gateway OPA sidecar when the gateway is enabled. The gateway OPA sidecar runs in the gateway pod, so it scales with the gateway replica count. +* The Operator automatically deploys the OAuth proxy when the Jaeger UI is enabled. The Jaeger UI and the OAuth proxy run in the query front end pod, so they scale with the query front end replica count. ==== When determining resources for a component, the Operator uses the following priority order: @@ -271,7 +342,7 @@ spec: ---- where: -`example`:: In this example, the following settings apply: The ingester component uses the explicitly defined resources, 4 CPU and 20 Gi memory, instead of the `1x.small` defaults. The querier component uses four replicas with resources from the `1x.small` profile. All the other components, such as the distributor, compactor, and query front end, use the `1x.small` default resources. +`example`:: In this example, the following settings apply: The ingester component uses the explicitly defined resources, 4 CPU and 20 Gi memory, instead of the `1x.small` defaults. The querier component uses 4 replicas instead of the 3 replicas of the `1x.small` profile, with resources from the `1x.small` profile. All the other components, such as the distributor, compactor, and query front end, use the `1x.small` default resources and replica counts. `size`:: Specifies the base size profile for all components. `resources`:: Overrides the ingester resources. The ingester component uses these explicit values instead of the `1x.small` defaults. -`replicas`:: Specifies the number of querier replicas. The replica counts are independent of the size profile. +`replicas`:: Overrides the number of querier replicas. This value takes precedence over the replica count of the size profile. diff --git a/modules/distr-tracing-tempo-config-zone-aware-replication.adoc b/modules/distr-tracing-tempo-config-zone-aware-replication.adoc new file mode 100644 index 00000000000..3a912584b0f --- /dev/null +++ b/modules/distr-tracing-tempo-config-zone-aware-replication.adoc @@ -0,0 +1,80 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: PROCEDURE +[id="distr-tracing-tempo-config-zone-aware-replication_{context}"] += Configuring zone-aware replication + +[role="_abstract"] +Configure the `spec.replicationZones` field to spread the TempoStack pods across availability zones and to replicate spans across those zones. + +.Prerequisites + +* You have installed the {TempoOperator}. +* The nodes of your cluster are labeled with the topology key that you intend to use, such as `topology.kubernetes.io/zone`. +* The number of distinct availability zones is greater than or equal to the replication factor that you intend to configure. + +.Procedure + +. Configure the `spec.replicationZones` field in your `TempoStack` custom resource (CR). ++ +This example `TempoStack` CR has zone-aware replication enabled: ++ +[source,yaml] +---- +apiVersion: tempo.grafana.com/v1alpha1 +kind: TempoStack +metadata: + name: example + namespace: tracing-system +spec: + size: 1x.small + replicationFactor: 3 + replicationZones: + - maxSkew: 1 + topologyKey: topology.kubernetes.io/zone + storage: + secret: + name: object-storage-secret + type: s3 + template: + ingester: + replicas: 3 +---- +where: + +`spec.replicationFactor`:: Specifies how many ingesters must acknowledge a span. This value must be less than or equal to the number of available zones. +`spec.replicationZones.maxSkew`:: Specifies the maximum difference in the number of pods of a component between any 2 topology domains. The default is 1, and you cannot specify a value lower than 1. +`spec.replicationZones.topologyKey`:: Specifies a zone in the form of a topology key that corresponds to a node label. +`spec.template.ingester.replicas`:: Specifies the number of ingester replicas. To place an ingester in each zone, set this value to the number of zones. + +. Apply the CR by running the following command: ++ +[source,terminal] +---- +$ oc apply -f .yaml -n tracing-system +---- + +.Verification + +. Verify that the pods of a component are distributed across the zones by running the following command: ++ +[source,terminal] +---- +$ oc get pods -n tracing-system \ + -l app.kubernetes.io/instance=example,app.kubernetes.io/component=ingester \ + -o custom-columns='NAME:.metadata.name,ZONE:.metadata.annotations.tempo\.grafana\.com/availability-zone' +---- ++ +Example output: ++ +[source,terminal] +---- +NAME ZONE +tempo-example-ingester-0 us-east-1a +tempo-example-ingester-1 us-east-1b +tempo-example-ingester-2 us-east-1c +---- + +. If a pod remains in the `Init` state, verify that the `tempo.grafana.com/availability-zone` annotation is populated. An empty annotation means that the node that the pod was scheduled on is missing the configured topology label. diff --git a/modules/distr-tracing-tempo-ha-component-replicas.adoc b/modules/distr-tracing-tempo-ha-component-replicas.adoc new file mode 100644 index 00000000000..c6f656b2fb8 --- /dev/null +++ b/modules/distr-tracing-tempo-ha-component-replicas.adoc @@ -0,0 +1,108 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: REFERENCE +[id="distr-tracing-tempo-ha-component-replicas_{context}"] += Default component replicas per deployment size + +[role="_abstract"] +The deployment size sets a default replica count for every TempoStack component, so that a single pod or node failure cannot break the ingest path or the query path. + +When you set the `spec.size` field to any size other than `1x.demo`, the {TempoOperator} defaults every component to at least 2 replicas and scales the throughput-bound components, which are the distributor, querier, and metrics generator, at the larger sizes. The following table shows the default replica count for each component at each deployment size: + +[cols="1h,5*",options="header"] +|=== +|Component +|`1x.demo` +|`1x.pico` +|`1x.extra-small` +|`1x.small` +|`1x.medium` + +|Distributor +|1 +|2 +|2 +|3 +|4 + +|Ingester +|1 +|2 +|2 +|2 +|2 + +|Querier +|1 +|2 +|2 +|3 +|5 + +|Query front end +|1 +|2 +|2 +|2 +|2 + +|Compactor +|1 +|2 +|2 +|2 +|3 + +|Gateway +|1 +|2 +|2 +|2 +|2 + +|Metrics generator +|1 +|2 +|2 +|3 +|3 +|=== + +[NOTE] +==== +* The ingester replica count is not taken from this profile: it is derived from the replication factor as `floor(replicationFactor / 2) + 1`. The values in the table are the counts that result from the default replication factor of each size. +* The gateway and metrics generator are optional components. Their replica counts apply only when you enable them. +* When you do not set the `spec.size` field, every component defaults to a single replica. +==== + +To override the default for a component, set `spec.template..replicas`. An explicit replica count always takes precedence over the size profile, and you can override individual components while the remaining components keep the defaults of the size. + +This example `TempoStack` custom resource sets a component replica override: + +[source,yaml] +---- +apiVersion: tempo.grafana.com/v1alpha1 +kind: TempoStack +metadata: + name: example + namespace: tracing-system +spec: + size: 1x.small + storage: + secret: + name: object-storage-secret + type: s3 + template: + querier: + replicas: 6 +---- +where: + +`replicas`:: Specifies 6 querier replicas instead of the 3 replicas of the `1x.small` size. All the other components keep the default replica counts of the `1x.small` size. + +[IMPORTANT] +==== +Each replica requests the resources that are configured for its component in the deployment size. Increasing a replica count therefore increases the CPU and memory that the TempoStack instance requests from the cluster. +==== diff --git a/modules/distr-tracing-tempo-ha-pod-disruption-budgets.adoc b/modules/distr-tracing-tempo-ha-pod-disruption-budgets.adoc new file mode 100644 index 00000000000..d3285fd2a71 --- /dev/null +++ b/modules/distr-tracing-tempo-ha-pod-disruption-budgets.adoc @@ -0,0 +1,37 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: CONCEPT +[id="distr-tracing-tempo-ha-pod-disruption-budgets_{context}"] += TempoStack behavior during cluster restarts + +[role="_abstract"] +Pod disruption budgets keep a TempoStack instance ingesting and querying traces while nodes are drained, for example during a cluster update. + +The {TempoOperator} creates a `PodDisruptionBudget` object for each of the following components of a `TempoStack` custom resource (CR): + +* Distributor +* Ingester +* Querier +* Query front end +* Gateway (when enabled) +* Metrics generator (when enabled) + +Each budget sets `maxUnavailable: 1`, so that a voluntary disruption, such as a node drain, can evict only 1 replica of a component at a time. The remaining replicas keep serving traffic, which means that there is no downtime for the TempoStack instance during {ocp-product-title} cluster updates. + +The Operator names each `PodDisruptionBudget` object after its component, in the format `tempo--`, and creates it in the namespace of the `TempoStack` CR. When you disable the gateway or the metrics generator, the Operator removes the corresponding budget. + +The command in this example lists the pod disruption budgets of a TempoStack instance: + +[source,terminal] +---- +$ oc get poddisruptionbudget -n tracing-system +---- + +[NOTE] +==== +* The Operator does not create a `PodDisruptionBudget` object for the compactor. +* Pod disruption budgets apply to the `TempoStack` CR only. The `TempoMonolithic` CR is unaffected. +* A pod disruption budget is only effective when the component runs more than 1 replica. On the `1x.demo` size, and when you do not set the `spec.size` field, each component runs a single replica, so a node drain interrupts that component. +==== diff --git a/modules/distr-tracing-tempo-ha-pod-placement.adoc b/modules/distr-tracing-tempo-ha-pod-placement.adoc new file mode 100644 index 00000000000..a637ffeaa18 --- /dev/null +++ b/modules/distr-tracing-tempo-ha-pod-placement.adoc @@ -0,0 +1,56 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: CONCEPT +[id="distr-tracing-tempo-ha-pod-placement_{context}"] += TempoStack pod placement + +[role="_abstract"] +Pod anti-affinity rules request that the scheduler place the pods of the same component on different nodes, so that the loss of a single node does not take down a whole component. + +The {TempoOperator} sets default preferred `podAntiAffinity` rules on all TempoStack components. The rules select pods of the same component and use the following topology keys, in order of decreasing weight: + +`kubernetes.io/hostname`:: + The node. This is the preferred rule, with a weight of 100, and it spreads the replicas of a component across different nodes. + +`failure-domain.beta.kubernetes.io/zone`:: + The failure domain of the node, with a weight of 75. + +Because these rules are preferred rather than required, the scheduler places a pod on a node that already runs another replica of the same component when no better node is available, instead of leaving the pod in the `Pending` state. To enforce placement across availability zones instead, configure zone-aware replication, which uses topology spread constraints with the `DoNotSchedule` policy. + +You cannot override the anti-affinity rules that the Operator sets. To restrict which nodes a component runs on, set the `spec.template..nodeSelector` and `spec.template..tolerations` fields. + +This example `TempoStack` custom resource sets a node selector and a toleration for the ingester: + +[source,yaml] +---- +apiVersion: tempo.grafana.com/v1alpha1 +kind: TempoStack +metadata: + name: example + namespace: tracing-system +spec: + size: 1x.small + storage: + secret: + name: object-storage-secret + type: s3 + template: + ingester: + nodeSelector: + node-role.kubernetes.io/infra: '' + tolerations: + - key: node-role.kubernetes.io/infra + operator: Exists + effect: NoSchedule +---- +where: + +`nodeSelector`:: Specifies the node labels that a node must have for the ingester pods to be scheduled on it. +`tolerations`:: Specifies the taints that the ingester pods tolerate. + +[IMPORTANT] +==== +Restricting a component to a small set of nodes can prevent the scheduler from satisfying the anti-affinity rules. Ensure that the selected nodes are at least as many as the replica count of the component. +==== diff --git a/modules/distr-tracing-tempo-ha-replication-factor.adoc b/modules/distr-tracing-tempo-ha-replication-factor.adoc new file mode 100644 index 00000000000..916da3ad2ac --- /dev/null +++ b/modules/distr-tracing-tempo-ha-replication-factor.adoc @@ -0,0 +1,65 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: CONCEPT +[id="distr-tracing-tempo-ha-replication-factor_{context}"] += Replication factor and ingester quorum + +[role="_abstract"] +The replication factor determines how many ingesters store a copy of each span, and therefore how many ingesters a TempoStack instance can lose without losing data that is not yet flushed to object storage. + +The `spec.replicationFactor` field defines how many ingesters must acknowledge a span before the distributor accepts it. When you set the `spec.size` field and do not set `spec.replicationFactor`, the {TempoOperator} defaults the replication factor from the size, as the following table shows: + +[cols="1h,5*",options="header"] +|=== +| +|`1x.demo` +|`1x.pico` +|`1x.extra-small` +|`1x.small` +|`1x.medium` + +|Replication factor +|1 +|2 +|2 +|2 +|2 +|=== + +When you do not set the `spec.size` field either, the replication factor defaults to 1, which provides no redundancy. + +The number of ingester replicas must be sufficient to form a quorum for the configured replication factor. The quorum is `floor(replicationFactor / 2) + 1`. The {TempoOperator} defaults `spec.template.ingester.replicas` to the quorum for the configured replication factor whenever you set the `spec.size` field and do not set the replica count yourself. If you set both fields and the ingester replica count is lower than the quorum, the Operator rejects the `TempoStack` custom resource (CR) with an error such as `replica factor of 3 requires at least 2 ingester replicas`. + +Setting the ingester replica count equal to the quorum is a valid configuration, but it leaves no room to lose an ingester. To tolerate the loss of ingesters, set `spec.template.ingester.replicas` higher than the quorum. + +This example `TempoStack` custom resource sets an explicit replication factor: + +[source,yaml] +---- +apiVersion: tempo.grafana.com/v1alpha1 +kind: TempoStack +metadata: + name: example + namespace: tracing-system +spec: + size: 1x.small + replicationFactor: 3 + storage: + secret: + name: object-storage-secret + type: s3 + template: + ingester: + replicas: 3 +---- +where: + +`replicationFactor`:: Specifies how many ingesters must acknowledge a span. This value overrides the default replication factor of the configured size. +`replicas`:: Specifies the number of ingester replicas. This value must be greater than or equal to the quorum, which is `floor(replicationFactor / 2) + 1`. + +[IMPORTANT] +==== +The replication factor protects the spans that ingesters hold in memory and in their write-ahead log before flushing them to object storage. It does not replace backups of the object storage bucket, where Tempo persists the trace blocks. +==== diff --git a/modules/distr-tracing-tempo-zone-fail-recovery.adoc b/modules/distr-tracing-tempo-zone-fail-recovery.adoc new file mode 100644 index 00000000000..bf1ce011557 --- /dev/null +++ b/modules/distr-tracing-tempo-zone-fail-recovery.adoc @@ -0,0 +1,87 @@ +// Module included in the following assemblies: +// +// * configuring/distr-tracing-tempo-high-availability.adoc + +:_mod-docs-content-type: PROCEDURE +[id="distr-tracing-tempo-zone-fail-recovery_{context}"] += Recovering TempoStack pods from failed zones + +[role="_abstract"] +Manually delete the persistent volume claims of the ingester pods in a failed zone to allow automatic rescheduling. + +In {ocp-product-title}, a zone failure happens when specific availability zone resources become inaccessible. If your {ocp-product-title} cluster is not configured to handle this, a zone failure can lead to service or data loss. + +The ingester pods of a TempoStack instance are part of a StatefulSet, and they come with persistent volume claims (PVCs) provisioned by a `StorageClass` object. Each ingester pod and its PVC reside in the same zone. When a zone failure occurs, the StatefulSet controller automatically attempts to recover the affected pods, but because the associated PVCs are also in the failed zone, automatic rescheduling to a different zone does not work. You must manually delete the PVCs in the failed zone to allow successful re-creation of the ingester pod and its provisioned PVC in the new zone. + +The other TempoStack components, such as the distributor, querier, query front end, compactor, and gateway, are stateless deployments without PVCs, and the scheduler reschedules them into a healthy zone without any manual intervention. + +[WARNING] +==== +You are about to delete the PVCs in the failed zone, and all data contained therein. The data to be deleted includes the spans that the affected ingesters have not yet flushed to object storage. The trace blocks that are already in object storage are unaffected. To avoid losing spans, always set the `spec.replicationFactor` field of the `TempoStack` custom resource (CR) to a value greater than 1, so that another ingester in a healthy zone holds a copy of each span. +==== + +.Prerequisites + +* The `TempoStack` CR has a replication factor greater than 1. +* A zone failure is detected by the control plane, and nodes in the failed zone are marked by cloud provider integration. + +.Procedure + +. List the pods in the `Pending` status by running the following command: ++ +[source,terminal] +---- +$ oc get pods --field-selector status.phase==Pending -n tracing-system +---- ++ +Example `oc get pods` output: ++ +[source,terminal] +---- +NAME READY STATUS RESTARTS AGE +tempo-example-ingester-1 0/1 Pending 0 16m +---- ++ +`STATUS`:: These pods are in the `Pending` status because their corresponding PVCs are in the failed zone. + +. List the PVCs in the `Pending` status by running the following command: ++ +[source,terminal] +---- +$ oc get pvc -o=json -n tracing-system | jq '.items[] | select(.status.phase == "Pending") | .metadata.name' -r +---- ++ +Example `oc get pvc` output: ++ +[source,terminal] +---- +data-tempo-example-ingester-1 +---- + +. Delete the PVC for a pod by running the following command: ++ +[source,terminal] +---- +$ oc delete pvc -n tracing-system +---- + +. Delete the pod by running the following command: ++ +[source,terminal] +---- +$ oc delete pod -n tracing-system +---- ++ +After these objects are successfully deleted, they are automatically rescheduled in an available zone. + +. Optional: If PVCs hang in the terminating state without being deleted, the PVC metadata finalizers might be set to `kubernetes.io/pv-protection`. Remove the finalizer for each PVC by running the following command, and then retry the deletion: ++ +[source,terminal] +---- +$ oc patch pvc -p '{"metadata":{"finalizers":null}}' -n tracing-system +---- + +[role="_additional-resources"] +.Additional resources + +* link:https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/[Kubernetes Documentation: StatefulSet]