[FAQ] Grafana alert provisioned from YAML: label shows "{{ .http_route }}" instead of the route #422

Course

ai-dev-tools-zoomcamp

Question

I provision a Grafana alert rule from a YAML file (Module 4, DevOps). A templated label such as route: "{{ $labels.http_route }}" does not get the route value. Every alert carries the literal text {{ .http_route }} instead. The same template works in the rule's annotations. Why?

Answer

Grafana's file provisioning expands environment variables ($VAR and ${VAR}) in provisioning files before the rule is created. In a rule's labels: block, $labels is treated as an environment variable named labels. It is not set, so it is replaced with nothing, and the rule is stored with {{ .http_route }}.

Escape the dollar sign with $$ in the labels: block:

labels:
  severity: critical
  service: order-tracker
  # `$$` escapes Grafana's env-var interpolation
  route: "{{ $$labels.http_route }}"
annotations:
  summary: "{{ $values.B.Value }} 5xx response(s) on {{ $labels.http_route }} in the last 1m"

Notes:

• Seen on Grafana 12.3.11 with a Grafana-managed rule provisioned from provisioning/alerting/*.yaml. Annotations were not affected there, so a single $ still works in them.
• This only applies to provisioning files. A rule created in the UI uses a single $.
• How to check: open the rule in Alerting > Alert rules and look at the stored label. If it reads {{ .http_route }}, the $labels was stripped.
• It matters when a notification policy or a webhook receiver routes on that label, because every alert then has the same value.

Working example: https://github.com/Sanjomwa/order-tracker/blob/main/observability/grafana/provisioning/alerting/order-tracker-5xx.yaml

Checklist

• I have searched existing FAQs and this question is not already answered
• The answer provides accurate, helpful information
• I have included any relevant code examples or links