feat(monitoring): dashboard and alert rules for iroh-dns-server

.claude/skills/grafana-dashboards/SKILL.md

@@ -27,8 +27,8 @@ four-dashboard suite and alert rules that go with them. `decdn/decdn` ships none
 their only home.
 
 **The files** (the node's in `monitoring/decdn-node/`, sponsord's in `monitoring/sponsord/`,
-the iroh relay's in `monitoring/iroh-relay/`, the public gateway's in
-`monitoring/decdn-gateway/`):
+the iroh relay's in `monitoring/iroh-relay/`, the iroh DNS server's in
+`monitoring/iroh-dns-server/`, the public gateway's in `monitoring/decdn-gateway/`):
 
 - `grafana-dashboard.json` — "deCDN — fleet overview", uid `decdn-poc-overview`.
   Fleet status, delivery funnel, slash safety, logs and traces.
@@ -67,6 +67,17 @@ the iroh relay's in `monitoring/iroh-relay/`, the public gateway's in
   Logs: `{unit="iroh-relay.service"}` with the parsed `level`. Unit tests in
   `prometheus-alerts_test.yml`.
 
+- `monitoring/iroh-dns-server/` — the self-hosted iroh DNS server's pair:
+  `dashboard-iroh-dns-server.json` (uid `decdn-iroh-dns-server`) and `prometheus-alerts.yml`
+  (group `iroh-dns-server`, `job="iroh-dns-server"`), Ansible and Compose paths, not rendered
+  by the chart. Unlabeled `dns_server_*` counters plus two gauges (`cache_zones`,
+  `cache_zones_dht`); TCP DNS counts in `dns_requests_total` only (tcp = total − udp −
+  https), and the mean HTTP duration is `http_requests_duration_ms_total` over
+  `http_requests_total` (no histogram). Name-checked in CI against `exported-metrics.txt`,
+  like the relay's: re-capture it on an `iroh_dns_server_version` bump (the command is in
+  its header). Logs: `{unit="iroh-dns-server.service"}` with the parsed `level`. Unit tests
+  in `prometheus-alerts_test.yml`.
+
 - `monitoring/decdn-gateway/` — the public gateway's pair: `dashboard-decdn-gateway.json`
   (uid `decdn-gateway`) and `prometheus-alerts.yml` (group `decdn-gateway`,
   `job="decdn-gateway"`), the Ansible path, not rendered by the chart. Nine unlabeled

.github/workflows/ci.yml

@@ -47,7 +47,7 @@ jobs:
               - '.github/workflows/ci.yml'
             # The chart shares the schema-key inventory and checker with molecule,
             # and renders monitoring/decdn-node/ through a symlink; lint-helm also
-            # checks monitoring/sponsord/ and monitoring/iroh-relay/. The decdn CLI
+            # checks the other monitoring/<service>/ pairs. The decdn CLI
             # it validates against is verified with the role's vendored KEYS.
             helm:
               - 'charts/**'

AGENTS.md

@@ -82,7 +82,7 @@ compose/                # one-host Docker Compose path (README.md)
   tests/*.jq            # lint-compose: bind-sources, invariants, inline-env, fail-closed
   tests/test_decdn_compose.py  # the wrapper's unit tests (make test-scripts)
 charts/decdn-node/      # Helm chart (README.md); ci/ values mirror molecule/schema's three plays; files/monitoring -> ../../../monitoring/decdn-node (symlink); tests/render-test.sh (make lint-helm); artifacthub-repo.yml
-monitoring/             # Grafana dashboards + Prometheus rules, maintained here by hand: decdn-node/ (rendered by the chart), sponsord/, iroh-relay/ (+ exported-metrics.txt); README.md
+monitoring/             # Grafana dashboards + Prometheus rules, maintained here by hand: decdn-node/ (rendered by the chart), sponsord/, iroh-relay/, iroh-dns-server/, decdn-gateway/ (the last three + exported-metrics.txt); README.md
 docs/                   # cross-path operator docs: node-operators.md, publishers.md, requirements.md, lifecycle.md
 examples/collection-fleet/  # consumer project for the collections, with its own CI, pre-commit and AGENTS.md; its inventory/ mirrors ansible/inventory/'s templates byte for byte except the hosts-*.yml.example comment headers (make test-scripts checks both parse to the same inventory): change both. release.sh moves its requirements.yml floors; its action SHAs and hook revs track this repo's (make test-scripts); galaxy/check-playbooks.sh runs it against the built collections
 scripts/                # upstream mirrors (sync-network-profiles.py, gen-schema-keys.py), render-compose-alloy.sh, the decdn bump (open-decdn-bump.sh, bump_decdn.py), fetch-decdn-cli.sh, release.sh (+ cliff.toml, chart-artifacthub-changes.py), molecule.sh, limit-guard.sh
@@ -136,10 +136,11 @@ and also publishes the binaries as crates (`decdn-node`, `decdn-cli`, `sponsord`
 `/metrics` and upstream's `adr/appendix-observability.md` registry, where a `planned`
 row emits nothing; decdn/decdn's `crates/sponsord/src/metrics.rs`) — nothing in CI
 checks those names, so a typo renders `(no data)` or a rule that never fires. The
-relay's `relayserver_*` and the gateway's `decdn_gateway_*` names *are* checked, against
-each one's `exported-metrics.txt`, the metrics of the pinned `iroh-relay` and decdn:
-re-capture the relay's when bumping `iroh_relay_version`, the gateway's when the decdn
-line moves.
+relay's `relayserver_*`, the DNS server's `dns_server_*` and the gateway's
+`decdn_gateway_*` names *are* checked, against each one's `exported-metrics.txt`, the
+metrics of the pinned `iroh-relay`, `iroh-dns-server` and decdn: re-capture the relay's
+when bumping `iroh_relay_version`, the DNS server's when bumping
+`iroh_dns_server_version`, the gateway's when the decdn line moves.
 The chart renders `monitoring/decdn-node/` through the `files/monitoring` symlink
 (`.Files` cannot read outside the chart), never through `tpl` (the alert annotations
 carry Prometheus templates), so copy a chart tree with `cp -RL`. Alert rules have
@@ -227,8 +228,9 @@ boundaries or exists nowhere else.
   stub is the holder; the role never touches the host's resolver.
 - A version bump (`iroh_relay_version`, `iroh_dns_server_version`, both tracking the
   iroh decdn builds against) moves every per-target sha256, the image digest and
-  version comment in `compose/compose.yaml`, `DNS_VERSION` in `decdn-compose` and the
-  relay's `exported-metrics.txt` (`make test-scripts` checks the names agree). The
+  version comment in `compose/compose.yaml` and `DNS_VERSION` in `decdn-compose`
+  (`make test-scripts` checks they agree with the role pins), and each one's
+  `exported-metrics.txt` (re-captured by hand; nothing checks its version header). The
   relay's allowlist play reads `decdn whoami` on every `decdn_nodes` host by
   `delegate_to`, so `LIMIT=<relay>` still needs those hosts reachable; relays deploy
   `serial: 1`.

README.md

@@ -167,7 +167,7 @@ This is the canonical statement; the per-path READMEs add only what is specific
 | [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data-node.yaml` (node operators: a cache node), `user-data-publisher.yaml` (publishers: an origin node, sponsord and its onramp), the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. |
 | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp`, an iroh relay, an iroh DNS server, a public gateway and Grafana Alloy, one profile each, and the `decdn-compose` wrapper. |
 | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. |
-| [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible and Compose). |
+| [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose), the iroh relay (`iroh-relay/`, Ansible and Compose), the iroh DNS server (`iroh-dns-server/`, Ansible and Compose) and the public gateway (`decdn-gateway/`, Ansible). |
 | [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. |
 | [`examples/collection-fleet/`](examples/collection-fleet/README.md) | A starting point for a private repository that deploys from the Galaxy collections alone, with only its own inventory. |
 | `scripts/` | Generators for the upstream mirrors (network profiles) and Compose's Alloy config (`render-compose-alloy.sh`), and the release gate. |

ansible/README.md

@@ -403,7 +403,8 @@ deploy-dns`; `make deploy` covers it too.
 - **One server per origin.** Upstream has no replication; the role README records
   the single point of failure as an open question.
 - **Observability:** with Grafana Cloud on, Alloy scrapes it on hosts in
-  `iroh_dns_server_hosts`. No dashboard or alert rules yet.
+  `iroh_dns_server_hosts`; the dashboard and alert rules are in
+  [`monitoring/iroh-dns-server/`](../monitoring/iroh-dns-server/).
 
 Setup and variables: [`roles/iroh_dns_server/README.md`](roles/iroh_dns_server/README.md).
 

ansible/roles/iroh_dns_server/README.md

@@ -224,9 +224,12 @@ then:
 - labels its journal stream `unit="iroh-dns-server.service"`, with a `level` parsed
   from each line.
 
-There is no dashboard or alert rule set in `monitoring/` for it yet; that is a
-follow-up. Certificate expiry is not exported. Watch `https://<hostname>/healthz`
-and an external `dig` with a check of your own.
+The dashboard and alert rules (down, lookup errors rising, publishes stopped) are
+in [`monitoring/iroh-dns-server/`](../../../monitoring/README.md#iroh-dns-server).
+Certificate expiry is not exported. Watch `https://<hostname>/healthz`, and DNS
+with an external `dig` (the server itself and the delegation, as
+[`monitoring/README.md`](../../../monitoring/README.md#iroh-dns-server) shows), with
+a check of your own.
 
 ## Day 2
 
@@ -302,3 +305,8 @@ and an external `dig` with a check of your own.
   - **The ACME certificate-cache name.** `warn` mode's escalation looks for
     tokio-rustls-acme 0.9's file name (`vars/main.yml`). If an iroh bump changes
     it, the escalation stops working without any error.
+  - **The exported metrics.** Re-capture
+    `monitoring/iroh-dns-server/exported-metrics.txt` from the new binary (the
+    command is in its header): `make lint-helm` then fails on a series the
+    dashboard or rules name that the new release dropped, and the gate's metric
+    (`vars/main.yml`) must still be in it.

ansible/roles/iroh_dns_server/vars/main.yml

@@ -31,5 +31,6 @@ _iroh_dns_server_archive_member: iroh-dns-server
 # hostname from a staging one (or one for an old name).
 _iroh_dns_server_acme_production_directory: https://acme-v02.api.letsencrypt.org/directory
 
-# One series the server always exports, for the readiness gate.
+# One series the server always exports, for the readiness gate (it is in
+# monitoring/iroh-dns-server/exported-metrics.txt).
 _iroh_dns_server_gate_metric: dns_server_dns_requests_total

charts/decdn-node/tests/render-test.sh

@@ -62,8 +62,9 @@ pass "schema-key checker: good/bad/empty fixtures"
 # The node's dashboards and rules live in the repo's monitoring/decdn-node/, which
 # operators on every deploy path import from; the chart reaches them through
 # files/monitoring, a relative symlink, because .Files cannot read outside the chart.
-# helm package must turn it into regular files, and nothing of sponsord's or
-# iroh-relay's may ship, nor artifacthub-repo.yml (release-chart.yml pushes it on its own).
+# helm package must turn it into regular files, and nothing of sponsord's,
+# iroh-relay's, iroh-dns-server's or decdn-gateway's may ship, nor
+# artifacthub-repo.yml (release-chart.yml pushes it on its own).
 link_target="../../../monitoring/decdn-node"
 # Prints what is wrong and returns non-zero, so the negatives below can call it too.
 check_monitoring_link() { # <chart dir> <monitoring dir> <empty scratch dir>
@@ -92,9 +93,9 @@ if not want:
 got = sorted(m.name[len(prefix):] for m in mon)
 if got != want:
     sys.exit(f"packaged files/monitoring {got}\n  != dashboards and rules in monitoring/decdn-node {want}")
-leak = [m.name for m in members if any(s in m.name.lower() for s in ("sponsord", "iroh-relay", "decdn-gateway"))]
+leak = [m.name for m in members if any(s in m.name.lower() for s in ("sponsord", "iroh-relay", "iroh-dns-server", "decdn-gateway"))]
 if leak:
-    sys.exit(f"sponsord, iroh-relay or decdn-gateway files in the chart package: {leak}")
+    sys.exit(f"sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package: {leak}")
 if any(m.name == "decdn-node/artifacthub-repo.yml" for m in members):
     sys.exit("artifacthub-repo.yml is in the chart package (.helmignore it; release-chart.yml pushes it on its own)")
 PY
@@ -137,13 +138,17 @@ touch "$neg/monitoring/decdn-node/README.md"
 expect_link_fail "a stray file in monitoring/decdn-node" "!= dashboards and rules"
 rm "$neg/monitoring/decdn-node/README.md"
 cp "$neg/monitoring/sponsord/dashboard-sponsord.json" "$neg/monitoring/decdn-node/"
-expect_link_fail "the sponsord dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package"
+expect_link_fail "the sponsord dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package"
 rm "$neg/monitoring/decdn-node/dashboard-sponsord.json"
 cp "$neg/monitoring/iroh-relay/dashboard-iroh-relay.json" "$neg/monitoring/decdn-node/"
-expect_link_fail "the iroh-relay dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package"
+expect_link_fail "the iroh-relay dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package"
 rm "$neg/monitoring/decdn-node/dashboard-iroh-relay.json"
 cp "$neg/monitoring/decdn-gateway/dashboard-decdn-gateway.json" "$neg/monitoring/decdn-node/"
-expect_link_fail "the decdn-gateway dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package"
+expect_link_fail "the decdn-gateway dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package"
+rm "$neg/monitoring/decdn-node/dashboard-decdn-gateway.json"
+cp "$neg/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json" "$neg/monitoring/decdn-node/"
+expect_link_fail "the iroh-dns-server dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package"
+rm "$neg/monitoring/decdn-node/dashboard-iroh-dns-server.json"
 
 # --- positive renders -----------------------------------------------------------
 for values in "$chart"/ci/*.yaml; do
@@ -272,21 +277,25 @@ if [ -n "$promtool" ]; then
   $promtool check rules < "$monitoring/decdn-gateway/prometheus-alerts.yml" > "$work/promtool.out" 2>&1 \
     || { cat "$work/promtool.out" >&2; fail "promtool check rules (monitoring/decdn-gateway/prometheus-alerts.yml)"; }
   pass "promtool check rules (decdn-gateway): $(grep -o '[0-9]* rules found' "$work/promtool.out")"
+  $promtool check rules < "$monitoring/iroh-dns-server/prometheus-alerts.yml" > "$work/promtool.out" 2>&1 \
+    || { cat "$work/promtool.out" >&2; fail "promtool check rules (monitoring/iroh-dns-server/prometheus-alerts.yml)"; }
+  pass "promtool check rules (iroh-dns-server): $(grep -o '[0-9]* rules found' "$work/promtool.out")"
 
   # The sponsord rules do time arithmetic on unix-time gauges, the iroh-relay
-  # rules divide one counter's rate by another's, and every *Down rule has an
+  # rules divide one counter's rate by another's, the iroh-dns-server rules compare
+  # a rate with its own over a day, and every *Down rule has an
   # absence arm, which only a series that stops can exercise. Only a unit test
   # catches mistakes in those.
   # promtool compares annotations exactly and they are prose, so each test runs
   # against a copy with them stripped (monitoring/<service>/prometheus-alerts_test.yml).
   mkdir "$work/rules-test"
-  for svc in decdn-node sponsord iroh-relay decdn-gateway; do
+  for svc in decdn-node sponsord iroh-relay iroh-dns-server decdn-gateway; do
     yq 'del(.groups[].rules[].annotations)' "$monitoring/$svc/prometheus-alerts.yml" \
       > "$work/rules-test/$svc-alerts.yml"
     cp "$monitoring/$svc/prometheus-alerts_test.yml" "$work/rules-test/$svc-alerts_test.yml"
   done
   chmod -R a+rX "$work/rules-test"
-  for svc in decdn-node sponsord iroh-relay decdn-gateway; do
+  for svc in decdn-node sponsord iroh-relay iroh-dns-server decdn-gateway; do
     if [ -n "${PROMTOOL_IMAGE:-}" ]; then
       docker run --rm -v "$work/rules-test:/w:ro" -w /w --entrypoint promtool "$PROMTOOL_IMAGE" \
         test rules "$svc-alerts_test.yml" > "$work/promtool.out" 2>&1
@@ -296,21 +305,21 @@ if [ -n "$promtool" ]; then
     pass "promtool test rules: $svc alerts"
   done
 else
-  skipped+=("promtool (PROMTOOL is empty): alert rule syntax and the decdn-node, sponsord, iroh-relay and decdn-gateway rule tests are NOT checked")
+  skipped+=("promtool (PROMTOOL is empty): alert rule syntax and the decdn-node, sponsord, iroh-relay, iroh-dns-server and decdn-gateway rule tests are NOT checked")
 fi
 
-# The sponsord, iroh-relay and decdn-gateway dashboards live outside the chart (monitoring/<service>/),
+# The sponsord, iroh-relay, iroh-dns-server and decdn-gateway dashboards live outside the chart (monitoring/<service>/),
 # so the chart never renders them (the monitoring-link check above keeps them out of
 # the package). They must still parse and carry a uid no other dashboard uses:
 # Grafana imports by uid.
-python3 - "$monitoring" <<'PY' || fail "sponsord / iroh-relay / decdn-gateway dashboards"
+python3 - "$monitoring" <<'PY' || fail "sponsord / iroh-relay / iroh-dns-server / decdn-gateway dashboards"
 import json, pathlib, sys
 root = pathlib.Path(sys.argv[1])
 uids = [json.load(open(p)).get("uid") for p in sorted((root / "decdn-node").glob("*.json"))]
 if not uids:
     sys.exit("no dashboard in monitoring/decdn-node/")
 dash = []
-for svc in ("sponsord", "iroh-relay", "decdn-gateway"):
+for svc in ("sponsord", "iroh-relay", "iroh-dns-server", "decdn-gateway"):
     found = sorted((root / svc).glob("*.json"))
     if not found:
         sys.exit(f"no dashboard in monitoring/{svc}/")
@@ -323,14 +332,14 @@ for p in dash:
 if len(uids) != len(set(uids)):
     sys.exit(f"duplicate dashboard uids: {sorted(uids)}")
 PY
-pass "sponsord, iroh-relay and decdn-gateway dashboards parse with their own uids"
+pass "sponsord, iroh-relay, iroh-dns-server and decdn-gateway dashboards parse with their own uids"
 
-# iroh-relay's and the gateway's metric names are checked against what the pinned
+# iroh-relay's, iroh-dns-server's and the gateway's metric names are checked against what the pinned
 # release exports (monitoring/<service>/exported-metrics.txt, captured from the real
 # binary): a typo would render "no data" or a rule that never fires. Every metric an
 # expr selects (an identifier before `{`) is checked, whatever its prefix, and so is
 # every series the rule tests feed in; only `up` comes from the scraper itself.
-for pair in iroh-relay:relayserver_ decdn-gateway:decdn_gateway_; do
+for pair in iroh-relay:relayserver_ iroh-dns-server:dns_server_ decdn-gateway:decdn_gateway_; do
   svc="${pair%%:*}" prefix="${pair#*:}"
   yq -o=json '.' "$monitoring/$svc/prometheus-alerts.yml" > "$work/$svc-alerts.json"
   yq -o=json '.' "$monitoring/$svc/prometheus-alerts_test.yml" > "$work/$svc-alerts-test.json"