[thiras]

Closes #118.

Adds monitoring/iroh-dns-server/, mirroring monitoring/iroh-relay/. Alloy already scrapes the server as job="iroh-dns-server" on the Ansible path and Compose's dns profile, and labels its journal unit="iroh-dns-server.service".

What's in it

• exported-metrics.txt: the /metrics of the pinned iroh-dns-server 1.3.0, captured from a fresh process. It was the x86_64 release asset, checked against the role's sha256. The header has the re-capture command. make lint-helm now checks dns_server_* names against it, as it does relayserver_*. This also pins the role's gate metric, dns_server_dns_requests_total.
• Dashboard (uid: decdn-iroh-dns-server):
  • stats: up (an instant query, as in feat(monitoring): scrape the public gateway, with its dashboard and alert rules #188), publishes and DNS requests per day, zones cached, mean HTTP duration;
  • pkarr publishes, update vs no-op;
  • the record store: inserted, updated, expired, removed;
  • cached zones;
  • DNS requests by transport (TCP is total − udp − https, because TCP increments only the total);
  • lookups: success, not found, error;
  • HTTP requests, 2xx vs non-2xx, and the mean duration;
  • the log.
• Rule group iroh-dns-server, with promtool unit tests:
  • IrohDnsServerDown: the *Down shape with the absence arm (monitoring: the *Down alerts never fire when the whole host dies (Ansible/Compose path) #109). The tests are the relay's block, relabelled.
  • IrohDnsServerLookupErrors: the 30m error rate is above 0.1/s and more than 3× the server's own 1d rate, for 15m.
  • IrohDnsServerNoPublishes: no pkarr put (update + no-op) in 30m after some in the last day, for 5m. Nodes republish every 5 minutes.
• Wiring and docs:
  • render-test.sh: promtool check and tests, the uid check, the name check, and the chart-leak substring and its negative.
  • Docs: monitoring/README.md gets a new section and the *Down list. In the role README, "no dashboard yet" is gone and the bump checklist gets a re-capture step. "No dashboard or alert rules yet" is gone from ansible/README.md and compose/README.md.
  • docs/lifecycle.md: silence the alerts before a migration or decommission. Repointing the nodes fires NoPublishes.
  • AGENTS.md and the grafana-dashboards skill.

Why the lookup-error rule is a rise, not a share

I ran the real 1.3.0 binary to see what moves dns_server_dns_lookup_error_total. It counts every answer other than NOERROR or NXDOMAIN:

• A non-query opcode (a scanner's UPDATE or STATUS probe) gets NOTIMP and counts as an error.
• A query for a name outside the origins is NXDOMAIN, because the server is also authoritative for ., so it counts as not found.
• A packet that does not parse is not counted.

So a public udp/53 has a steady floor of errors, and a quiet server could sit above any fixed share for good. Comparing against the server's own day absorbs that floor. The 0.1/s floor keeps a server with no baseline from firing on a handful of probes. The rule comment documents one limit: a server less than a day old has an understated baseline, and a unit test pins that.

Certificate expiry is not exported. The docs point to an external https://<hostname>/healthz probe and an external dig, as the relay's do.

Verification

• make lint-helm: promtool check (3 rules), the unit tests, the uid check, and "iroh-dns-server dashboard and rules name only exported series" all pass. The new chart-leak negative also passes.
• Negatives, not committed:
  • dns_server_bogus_total in the dashboard → the name check fails.
  • A rule threshold of 30× → the unit tests fail.
  • Publishes that resume → the unit tests fail.
• make lint, make test-scripts: pass.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

Dashboard and alert descriptions misrepresent packet acceptance, lookup-cache occupancy, and store retention, potentially misleading operators about address churn, published-node counts, and cached-answer expiry.

3 open issues

• Gauge measures lookup-cache occupancy, not published nodes or records · New
• Accepted packets may be counted as updates when only timestamps change · New
• Store retention does not guarantee cached answers expire after seven days · New

What changed in this PR

Adds monitoring for self-hosted iroh DNS servers, using the existing Alloy metrics and logs on Ansible and Compose deployments.

Changes:

• Adds a dashboard covering publishes, DNS lookups, HTTP traffic, cache activity, and logs.
• Adds three alert rules with unit tests and an exported-metrics snapshot.
• Extends validation and documents monitoring setup, maintenance, and alert silencing.

File	Description
monitoring/​README.md	Documents DNS monitoring assets and external probes.
monitoring/​iroh-dns-server/​prometheus-alerts.yml	Adds downtime, lookup-error, and stopped-publish alerts.
monitoring/​iroh-dns-server/​prometheus-alerts_test.yml	Tests alert timing and representative failure scenarios.
monitoring/​iroh-dns-server/​exported-metrics.txt	Records the pinned server’s exported metrics.
monitoring/​iroh-dns-server/​dashboard-iroh-dns-server.json	Adds the DNS server dashboard.
docs/​lifecycle.md	Documents silencing alerts during migration and decommission.
compose/​README.md	Links monitoring assets and describes Alloy integration.
charts/​decdn-node/​tests/​render-test.sh	Extends rule, metric-name, dashboard, and package-isolation checks.
ansible/​roles/​iroh_dns_server/​vars/​main.yml	Adds a comment linking the readiness metric to the snapshot.
ansible/​roles/​iroh_dns_server/​README.md	Links monitoring assets and adds metric recapture guidance.
ansible/​README.md	Links the new dashboard and alerts.
AGENTS.md	Updates the monitoring map and release-maintenance guidance.
.github/​workflows/​ci.yml	Clarifies monitoring coverage in a comment.
.claude/​skills/​grafana-dashboards/​SKILL.md	Documents DNS dashboard metrics and maintenance.

_{Review effort: Balanced}