From c0c4a727482c0d009235a384257280162992d6f9 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sun, 1 Nov 2026 15:34:53 +0300 Subject: [PATCH 1/3] feat(monitoring): dashboard and alert rules for iroh-dns-server - monitoring/iroh-dns-server/exported-metrics.txt: the /metrics of the pinned iroh-dns-server 1.3.0 (x86_64 release asset, sha256-checked), captured from a fresh process. make lint-helm checks every dns_server_* name against it, as it does relayserver_* (this also pins the role's readiness-gate metric). - dashboard-iroh-dns-server.json (uid decdn-iroh-dns-server): up (instant), publishes and DNS requests per day, zones cached, mean HTTP duration; pkarr publishes (update vs no-op), the record store, cached zones; DNS requests by transport (tcp = total - udp - https), lookups by answer; HTTP requests and mean duration; the server's log. - Rule group iroh-dns-server with promtool tests: IrohDnsServerDown (the *Down shape with the absence arm), IrohDnsServerLookupErrors (a rise against the server's own last day with a 0.1/s floor: a scanner's NOTIMP counts as an error, an out-of-zone query is NXDOMAIN, both checked on the real binary), IrohDnsServerNoPublishes (no pkarr put in 30m after some in the last day). - render-test.sh, docs (monitoring/README, the role README and its bump checklist, ansible/ and compose/ READMEs, lifecycle silence notes, AGENTS.md, the grafana-dashboards skill). Closes #118 Co-Authored-By: Claude Opus 5.5 --- .claude/skills/grafana-dashboards/SKILL.md | 15 +- .github/workflows/ci.yml | 2 +- AGENTS.md | 15 +- ansible/README.md | 3 +- ansible/roles/iroh_dns_server/README.md | 12 +- ansible/roles/iroh_dns_server/vars/main.yml | 3 +- charts/decdn-node/tests/render-test.sh | 41 +- compose/README.md | 9 +- docs/lifecycle.md | 8 +- monitoring/README.md | 43 +- .../dashboard-iroh-dns-server.json | 1073 +++++++++++++++++ .../iroh-dns-server/exported-metrics.txt | 63 + .../iroh-dns-server/prometheus-alerts.yml | 129 ++ .../prometheus-alerts_test.yml | 159 +++ 14 files changed, 1534 insertions(+), 41 deletions(-) create mode 100644 monitoring/iroh-dns-server/dashboard-iroh-dns-server.json create mode 100644 monitoring/iroh-dns-server/exported-metrics.txt create mode 100644 monitoring/iroh-dns-server/prometheus-alerts.yml create mode 100644 monitoring/iroh-dns-server/prometheus-alerts_test.yml diff --git a/.claude/skills/grafana-dashboards/SKILL.md b/.claude/skills/grafana-dashboards/SKILL.md index 6fe11f18..617bc889 100644 --- a/.claude/skills/grafana-dashboards/SKILL.md +++ b/.claude/skills/grafana-dashboards/SKILL.md @@ -27,8 +27,8 @@ four-dashboard suite and alert rules that go with them. `decdn/decdn` ships none their only home. **The files** (the node's in `monitoring/decdn-node/`, sponsord's in `monitoring/sponsord/`, -the iroh relay's in `monitoring/iroh-relay/`, the public gateway's in -`monitoring/decdn-gateway/`): +the iroh relay's in `monitoring/iroh-relay/`, the iroh DNS server's in +`monitoring/iroh-dns-server/`, the public gateway's in `monitoring/decdn-gateway/`): - `grafana-dashboard.json` — "deCDN — fleet overview", uid `decdn-poc-overview`. Fleet status, delivery funnel, slash safety, logs and traces. @@ -67,6 +67,17 @@ the iroh relay's in `monitoring/iroh-relay/`, the public gateway's in Logs: `{unit="iroh-relay.service"}` with the parsed `level`. Unit tests in `prometheus-alerts_test.yml`. +- `monitoring/iroh-dns-server/` — the self-hosted iroh DNS server's pair: + `dashboard-iroh-dns-server.json` (uid `decdn-iroh-dns-server`) and `prometheus-alerts.yml` + (group `iroh-dns-server`, `job="iroh-dns-server"`), Ansible and Compose paths, not rendered + by the chart. Unlabeled `dns_server_*` counters plus two gauges (`cache_zones`, + `cache_zones_dht`); TCP DNS counts in `dns_requests_total` only (tcp = total − udp − + https), and the mean HTTP duration is `http_requests_duration_ms_total` over + `http_requests_total` (no histogram). Name-checked in CI against `exported-metrics.txt`, + like the relay's: re-capture it on an `iroh_dns_server_version` bump (the command is in + its header). Logs: `{unit="iroh-dns-server.service"}` with the parsed `level`. Unit tests + in `prometheus-alerts_test.yml`. + - `monitoring/decdn-gateway/` — the public gateway's pair: `dashboard-decdn-gateway.json` (uid `decdn-gateway`) and `prometheus-alerts.yml` (group `decdn-gateway`, `job="decdn-gateway"`), the Ansible path, not rendered by the chart. Nine unlabeled diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 22647551..7c59b175 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -47,7 +47,7 @@ jobs: - '.github/workflows/ci.yml' # The chart shares the schema-key inventory and checker with molecule, # and renders monitoring/decdn-node/ through a symlink; lint-helm also - # checks monitoring/sponsord/ and monitoring/iroh-relay/. The decdn CLI + # checks the other monitoring// pairs. The decdn CLI # it validates against is verified with the role's vendored KEYS. helm: - 'charts/**' diff --git a/AGENTS.md b/AGENTS.md index bd837be2..db7ace44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -82,7 +82,7 @@ compose/ # one-host Docker Compose path (README.md) tests/*.jq # lint-compose: bind-sources, invariants, inline-env, fail-closed tests/test_decdn_compose.py # the wrapper's unit tests (make test-scripts) charts/decdn-node/ # Helm chart (README.md); ci/ values mirror molecule/schema's three plays; files/monitoring -> ../../../monitoring/decdn-node (symlink); tests/render-test.sh (make lint-helm); artifacthub-repo.yml -monitoring/ # Grafana dashboards + Prometheus rules, maintained here by hand: decdn-node/ (rendered by the chart), sponsord/, iroh-relay/ (+ exported-metrics.txt); README.md +monitoring/ # Grafana dashboards + Prometheus rules, maintained here by hand: decdn-node/ (rendered by the chart), sponsord/, iroh-relay/, iroh-dns-server/, decdn-gateway/ (the last three + exported-metrics.txt); README.md docs/ # cross-path operator docs: node-operators.md, publishers.md, requirements.md, lifecycle.md examples/collection-fleet/ # consumer project for the collections, with its own CI, pre-commit and AGENTS.md; its inventory/ mirrors ansible/inventory/'s templates byte for byte except the hosts-*.yml.example comment headers (make test-scripts checks both parse to the same inventory): change both. release.sh moves its requirements.yml floors; its action SHAs and hook revs track this repo's (make test-scripts); galaxy/check-playbooks.sh runs it against the built collections scripts/ # upstream mirrors (sync-network-profiles.py, gen-schema-keys.py), render-compose-alloy.sh, the decdn bump (open-decdn-bump.sh, bump_decdn.py), fetch-decdn-cli.sh, release.sh (+ cliff.toml, chart-artifacthub-changes.py), molecule.sh, limit-guard.sh @@ -136,10 +136,11 @@ and also publishes the binaries as crates (`decdn-node`, `decdn-cli`, `sponsord` `/metrics` and upstream's `adr/appendix-observability.md` registry, where a `planned` row emits nothing; decdn/decdn's `crates/sponsord/src/metrics.rs`) — nothing in CI checks those names, so a typo renders `(no data)` or a rule that never fires. The -relay's `relayserver_*` and the gateway's `decdn_gateway_*` names *are* checked, against -each one's `exported-metrics.txt`, the metrics of the pinned `iroh-relay` and decdn: -re-capture the relay's when bumping `iroh_relay_version`, the gateway's when the decdn -line moves. +relay's `relayserver_*`, the DNS server's `dns_server_*` and the gateway's +`decdn_gateway_*` names *are* checked, against each one's `exported-metrics.txt`, the +metrics of the pinned `iroh-relay`, `iroh-dns-server` and decdn: re-capture the relay's +when bumping `iroh_relay_version`, the DNS server's when bumping +`iroh_dns_server_version`, the gateway's when the decdn line moves. The chart renders `monitoring/decdn-node/` through the `files/monitoring` symlink (`.Files` cannot read outside the chart), never through `tpl` (the alert annotations carry Prometheus templates), so copy a chart tree with `cp -RL`. Alert rules have @@ -227,8 +228,8 @@ boundaries or exists nowhere else. stub is the holder; the role never touches the host's resolver. - A version bump (`iroh_relay_version`, `iroh_dns_server_version`, both tracking the iroh decdn builds against) moves every per-target sha256, the image digest and - version comment in `compose/compose.yaml`, `DNS_VERSION` in `decdn-compose` and the - relay's `exported-metrics.txt` (`make test-scripts` checks the names agree). The + version comment in `compose/compose.yaml`, `DNS_VERSION` in `decdn-compose` and each + one's `exported-metrics.txt` (`make test-scripts` checks the names agree). The relay's allowlist play reads `decdn whoami` on every `decdn_nodes` host by `delegate_to`, so `LIMIT=` still needs those hosts reachable; relays deploy `serial: 1`. diff --git a/ansible/README.md b/ansible/README.md index efd31288..79259806 100644 --- a/ansible/README.md +++ b/ansible/README.md @@ -403,7 +403,8 @@ deploy-dns`; `make deploy` covers it too. - **One server per origin.** Upstream has no replication; the role README records the single point of failure as an open question. - **Observability:** with Grafana Cloud on, Alloy scrapes it on hosts in - `iroh_dns_server_hosts`. No dashboard or alert rules yet. + `iroh_dns_server_hosts`; the dashboard and alert rules are in + [`monitoring/iroh-dns-server/`](../monitoring/iroh-dns-server/). Setup and variables: [`roles/iroh_dns_server/README.md`](roles/iroh_dns_server/README.md). diff --git a/ansible/roles/iroh_dns_server/README.md b/ansible/roles/iroh_dns_server/README.md index 8f0e4a48..f5d54030 100644 --- a/ansible/roles/iroh_dns_server/README.md +++ b/ansible/roles/iroh_dns_server/README.md @@ -224,9 +224,10 @@ then: - labels its journal stream `unit="iroh-dns-server.service"`, with a `level` parsed from each line. -There is no dashboard or alert rule set in `monitoring/` for it yet; that is a -follow-up. Certificate expiry is not exported. Watch `https:///healthz` -and an external `dig` with a check of your own. +The dashboard and alert rules (down, lookup errors rising, publishes stopped) are +in [`monitoring/iroh-dns-server/`](../../../monitoring/README.md#iroh-dns-server). +Certificate expiry is not exported. Watch `https:///healthz` and an +external `dig` with a check of your own. ## Day 2 @@ -302,3 +303,8 @@ and an external `dig` with a check of your own. - **The ACME certificate-cache name.** `warn` mode's escalation looks for tokio-rustls-acme 0.9's file name (`vars/main.yml`). If an iroh bump changes it, the escalation stops working without any error. + - **The exported metrics.** Re-capture + `monitoring/iroh-dns-server/exported-metrics.txt` from the new binary (the + command is in its header): `make lint-helm` then fails on a series the + dashboard or rules name that the new release dropped, and the gate's metric + (`vars/main.yml`) must still be in it. diff --git a/ansible/roles/iroh_dns_server/vars/main.yml b/ansible/roles/iroh_dns_server/vars/main.yml index e92f282f..d592b607 100644 --- a/ansible/roles/iroh_dns_server/vars/main.yml +++ b/ansible/roles/iroh_dns_server/vars/main.yml @@ -31,5 +31,6 @@ _iroh_dns_server_archive_member: iroh-dns-server # hostname from a staging one (or one for an old name). _iroh_dns_server_acme_production_directory: https://acme-v02.api.letsencrypt.org/directory -# One series the server always exports, for the readiness gate. +# One series the server always exports, for the readiness gate (it is in +# monitoring/iroh-dns-server/exported-metrics.txt). _iroh_dns_server_gate_metric: dns_server_dns_requests_total diff --git a/charts/decdn-node/tests/render-test.sh b/charts/decdn-node/tests/render-test.sh index 42a5dd82..14073574 100755 --- a/charts/decdn-node/tests/render-test.sh +++ b/charts/decdn-node/tests/render-test.sh @@ -62,8 +62,8 @@ pass "schema-key checker: good/bad/empty fixtures" # The node's dashboards and rules live in the repo's monitoring/decdn-node/, which # operators on every deploy path import from; the chart reaches them through # files/monitoring, a relative symlink, because .Files cannot read outside the chart. -# helm package must turn it into regular files, and nothing of sponsord's or -# iroh-relay's may ship, nor artifacthub-repo.yml (release-chart.yml pushes it on its own). +# helm package must turn it into regular files, and nothing of sponsord's, +# iroh-relay's, iroh-dns-server's or decdn-gateway's may ship, nor artifacthub-repo.yml (release-chart.yml pushes it on its own). link_target="../../../monitoring/decdn-node" # Prints what is wrong and returns non-zero, so the negatives below can call it too. check_monitoring_link() { # @@ -92,9 +92,9 @@ if not want: got = sorted(m.name[len(prefix):] for m in mon) if got != want: sys.exit(f"packaged files/monitoring {got}\n != dashboards and rules in monitoring/decdn-node {want}") -leak = [m.name for m in members if any(s in m.name.lower() for s in ("sponsord", "iroh-relay", "decdn-gateway"))] +leak = [m.name for m in members if any(s in m.name.lower() for s in ("sponsord", "iroh-relay", "iroh-dns-server", "decdn-gateway"))] if leak: - sys.exit(f"sponsord, iroh-relay or decdn-gateway files in the chart package: {leak}") + sys.exit(f"sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package: {leak}") if any(m.name == "decdn-node/artifacthub-repo.yml" for m in members): sys.exit("artifacthub-repo.yml is in the chart package (.helmignore it; release-chart.yml pushes it on its own)") PY @@ -137,13 +137,16 @@ touch "$neg/monitoring/decdn-node/README.md" expect_link_fail "a stray file in monitoring/decdn-node" "!= dashboards and rules" rm "$neg/monitoring/decdn-node/README.md" cp "$neg/monitoring/sponsord/dashboard-sponsord.json" "$neg/monitoring/decdn-node/" -expect_link_fail "the sponsord dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package" +expect_link_fail "the sponsord dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package" rm "$neg/monitoring/decdn-node/dashboard-sponsord.json" cp "$neg/monitoring/iroh-relay/dashboard-iroh-relay.json" "$neg/monitoring/decdn-node/" -expect_link_fail "the iroh-relay dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package" +expect_link_fail "the iroh-relay dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package" rm "$neg/monitoring/decdn-node/dashboard-iroh-relay.json" cp "$neg/monitoring/decdn-gateway/dashboard-decdn-gateway.json" "$neg/monitoring/decdn-node/" -expect_link_fail "the decdn-gateway dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay or decdn-gateway files in the chart package" +expect_link_fail "the decdn-gateway dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package" +rm "$neg/monitoring/decdn-node/dashboard-decdn-gateway.json" +cp "$neg/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json" "$neg/monitoring/decdn-node/" +expect_link_fail "the iroh-dns-server dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package" # --- positive renders ----------------------------------------------------------- for values in "$chart"/ci/*.yaml; do @@ -272,21 +275,25 @@ if [ -n "$promtool" ]; then $promtool check rules < "$monitoring/decdn-gateway/prometheus-alerts.yml" > "$work/promtool.out" 2>&1 \ || { cat "$work/promtool.out" >&2; fail "promtool check rules (monitoring/decdn-gateway/prometheus-alerts.yml)"; } pass "promtool check rules (decdn-gateway): $(grep -o '[0-9]* rules found' "$work/promtool.out")" + $promtool check rules < "$monitoring/iroh-dns-server/prometheus-alerts.yml" > "$work/promtool.out" 2>&1 \ + || { cat "$work/promtool.out" >&2; fail "promtool check rules (monitoring/iroh-dns-server/prometheus-alerts.yml)"; } + pass "promtool check rules (iroh-dns-server): $(grep -o '[0-9]* rules found' "$work/promtool.out")" # The sponsord rules do time arithmetic on unix-time gauges, the iroh-relay - # rules divide one counter's rate by another's, and every *Down rule has an + # rules divide one counter's rate by another's, the iroh-dns-server rules compare + # a rate with its own over a day, and every *Down rule has an # absence arm, which only a series that stops can exercise. Only a unit test # catches mistakes in those. # promtool compares annotations exactly and they are prose, so each test runs # against a copy with them stripped (monitoring//prometheus-alerts_test.yml). mkdir "$work/rules-test" - for svc in decdn-node sponsord iroh-relay decdn-gateway; do + for svc in decdn-node sponsord iroh-relay iroh-dns-server decdn-gateway; do yq 'del(.groups[].rules[].annotations)' "$monitoring/$svc/prometheus-alerts.yml" \ > "$work/rules-test/$svc-alerts.yml" cp "$monitoring/$svc/prometheus-alerts_test.yml" "$work/rules-test/$svc-alerts_test.yml" done chmod -R a+rX "$work/rules-test" - for svc in decdn-node sponsord iroh-relay decdn-gateway; do + for svc in decdn-node sponsord iroh-relay iroh-dns-server decdn-gateway; do if [ -n "${PROMTOOL_IMAGE:-}" ]; then docker run --rm -v "$work/rules-test:/w:ro" -w /w --entrypoint promtool "$PROMTOOL_IMAGE" \ test rules "$svc-alerts_test.yml" > "$work/promtool.out" 2>&1 @@ -296,21 +303,21 @@ if [ -n "$promtool" ]; then pass "promtool test rules: $svc alerts" done else - skipped+=("promtool (PROMTOOL is empty): alert rule syntax and the decdn-node, sponsord, iroh-relay and decdn-gateway rule tests are NOT checked") + skipped+=("promtool (PROMTOOL is empty): alert rule syntax and the decdn-node, sponsord, iroh-relay, iroh-dns-server and decdn-gateway rule tests are NOT checked") fi -# The sponsord, iroh-relay and decdn-gateway dashboards live outside the chart (monitoring//), +# The sponsord, iroh-relay, iroh-dns-server and decdn-gateway dashboards live outside the chart (monitoring//), # so the chart never renders them (the monitoring-link check above keeps them out of # the package). They must still parse and carry a uid no other dashboard uses: # Grafana imports by uid. -python3 - "$monitoring" <<'PY' || fail "sponsord / iroh-relay / decdn-gateway dashboards" +python3 - "$monitoring" <<'PY' || fail "sponsord / iroh-relay / iroh-dns-server / decdn-gateway dashboards" import json, pathlib, sys root = pathlib.Path(sys.argv[1]) uids = [json.load(open(p)).get("uid") for p in sorted((root / "decdn-node").glob("*.json"))] if not uids: sys.exit("no dashboard in monitoring/decdn-node/") dash = [] -for svc in ("sponsord", "iroh-relay", "decdn-gateway"): +for svc in ("sponsord", "iroh-relay", "iroh-dns-server", "decdn-gateway"): found = sorted((root / svc).glob("*.json")) if not found: sys.exit(f"no dashboard in monitoring/{svc}/") @@ -323,14 +330,14 @@ for p in dash: if len(uids) != len(set(uids)): sys.exit(f"duplicate dashboard uids: {sorted(uids)}") PY -pass "sponsord, iroh-relay and decdn-gateway dashboards parse with their own uids" +pass "sponsord, iroh-relay, iroh-dns-server and decdn-gateway dashboards parse with their own uids" -# iroh-relay's and the gateway's metric names are checked against what the pinned +# iroh-relay's, iroh-dns-server's and the gateway's metric names are checked against what the pinned # release exports (monitoring//exported-metrics.txt, captured from the real # binary): a typo would render "no data" or a rule that never fires. Every metric an # expr selects (an identifier before `{`) is checked, whatever its prefix, and so is # every series the rule tests feed in; only `up` comes from the scraper itself. -for pair in iroh-relay:relayserver_ decdn-gateway:decdn_gateway_; do +for pair in iroh-relay:relayserver_ iroh-dns-server:dns_server_ decdn-gateway:decdn_gateway_; do svc="${pair%%:*}" prefix="${pair#*:}" yq -o=json '.' "$monitoring/$svc/prometheus-alerts.yml" > "$work/$svc-alerts.json" yq -o=json '.' "$monitoring/$svc/prometheus-alerts_test.yml" > "$work/$svc-alerts-test.json" diff --git a/compose/README.md b/compose/README.md index 9b057b6c..61cb4f94 100644 --- a/compose/README.md +++ b/compose/README.md @@ -640,9 +640,10 @@ no longer insists on the pinned version). (iroh-dns-server ignores a mistyped key) and refuses to restart on a problem. - **Backup:** `backup` takes the config only. Nodes republish their records every 5 minutes, and the server gets a new certificate on a new host. -- **Monitoring:** no dashboard or alert rules yet. Its metrics are on - `127.0.0.1:9117/metrics`, which the [`alloy` profile](#grafana-cloud-alloy) on the - same host scrapes as `job="iroh-dns-server"`. +- **Dashboard and alerts:** in + [`monitoring/iroh-dns-server/`](../monitoring/README.md#iroh-dns-server). The + [`alloy` profile](#grafana-cloud-alloy) on the same host scrapes + `127.0.0.1:9117/metrics` as `job="iroh-dns-server"`; otherwise the scrape is your own. ## A public gateway (publishers) @@ -786,7 +787,7 @@ plus `region` and `deployment_environment` from `.env`): relay's (`127.0.0.1:9092`, `job="iroh-relay"`) with `relay`, and the DNS server's (`127.0.0.1:9117`, `job="iroh-dns-server"`) with `dns`. A host without the daemon has no target, so no `up == 0` for `DecdnNodeDown` (or `SponsordDown`, - `IrohRelayDown`) to fire on. Compose passes `COMPOSE_PROFILES` in as `DECDN_COMPOSE_PROFILES`, so a profile + `IrohRelayDown`, `IrohDnsServerDown`) to fire on. Compose passes `COMPOSE_PROFILES` in as `DECDN_COMPOSE_PROFILES`, so a profile change recreates the container with the new targets (a profile chosen with `--profile` on the command line instead is not seen); - the host: Alloy's in-process node_exporter (CPU, memory, disks, filesystems, diff --git a/docs/lifecycle.md b/docs/lifecycle.md index cb7e61e7..4ae06e77 100644 --- a/docs/lifecycle.md +++ b/docs/lifecycle.md @@ -368,7 +368,9 @@ and certificate, which a new host re-issues for itself. zone's glue record. Until the glue moves, resolvers and Let's Encrypt still reach the old host, and the deploy's certificate check warns; re-run `make deploy-dns LIMIT=` once it has moved. Nodes keep their `decdn_discovery_pkarr_url`. - Mind Let's Encrypt's rate limits as for the relay. + Mind Let's Encrypt's rate limits as for the relay. Once the publishes follow the + name, the old host fires `IrohDnsServerNoPublishes`: silence it there, with + `IrohDnsServerDown` before you take the old host down. - **Decommission:** point every node's `decdn_discovery_pkarr_url` and `decdn_discovery_dns_origin` elsewhere (or clear them, back to n0's discovery) and re-deploy them first: peers resolving under the origin find nothing once the @@ -377,7 +379,9 @@ and certificate, which a new host re-issues for itself. restart-inputs record, and tears down the Alloy agent. It keeps the binary, the config and the state directory. tcp/443 and udp+tcp/53 stay open until the host leaves `iroh_dns_server_hosts` and baseline runs again. Remove the delegation (NS and glue) - at the parent zone afterwards. + at the parent zone afterwards. Silence `IrohDnsServerNoPublishes` and + `IrohDnsServerDown` for the host before you re-point the nodes: both fire on a + server that went quiet, for up to a day. ## Public gateway diff --git a/monitoring/README.md b/monitoring/README.md index c4adafdf..5e5179c3 100644 --- a/monitoring/README.md +++ b/monitoring/README.md @@ -2,8 +2,9 @@ The deCDN Grafana dashboards and reference Prometheus alert rules, maintained here: `decdn-node/` for the node, on every deploy path, `sponsord/` for the onboarding -sponsor and `iroh-relay/` for the self-hosted iroh relay, both on the Ansible and -Compose paths, and `decdn-gateway/` for the public gateway, on the Ansible path. The node's metric surface is documented in +sponsor, `iroh-relay/` for the self-hosted iroh relay and `iroh-dns-server/` for the +self-hosted iroh DNS server, all three on the Ansible and Compose paths, and +`decdn-gateway/` for the public gateway, on the Ansible path. The node's metric surface is documented in upstream `decdn/decdn`'s `adr/appendix-observability.md`. The node's assets were imported from `decdn/decdn` (MIT OR Apache-2.0) at `20db95ef`; distributed here under this repo's MIT license. sponsord's were written here. @@ -18,6 +19,7 @@ this repo's MIT license. sponsord's were written here. | `decdn-node/prometheus-alerts_test.yml` | promtool unit tests for `DecdnNodeDown`, `DecdnCapacityBondRegistryResyncFailing`, `DecdnBuyerTopupUnresolved`, `DecdnRedemptionMaybeBroadcast`, `DecdnNodeClockSkew`, `DecdnSettlementLedgerWriteFailing`, `DecdnSettlementReanchorBacklog` and `DecdnSettlementReanchorLowerDeposit` (run by `make lint-helm`; the chart's `.helmignore` keeps them out of the package). | | `sponsord/` | The onboarding sponsor's dashboard and alert rules ([below](#sponsord)). Not rendered by the chart. | | `iroh-relay/` | The iroh relay's dashboard and alert rules ([below](#iroh-relay)). Not rendered by the chart. | +| `iroh-dns-server/` | The iroh DNS server's dashboard and alert rules ([below](#iroh-dns-server)). Not rendered by the chart. | | `decdn-gateway/` | The public gateway's dashboard and alert rules ([below](#public-gateway)). Not rendered by the chart. | **Editing.** Every `decdn_*` series a panel or rule names must be one `decdn-node` @@ -34,7 +36,7 @@ that `.helmignore`. ## The `*Down` alerts -`DecdnNodeDown`, `SponsordDown`, `IrohRelayDown` and `DecdnGatewayDown` fire on `up == 0`, and also on an +`DecdnNodeDown`, `SponsordDown`, `IrohRelayDown`, `IrohDnsServerDown` and `DecdnGatewayDown` fire on `up == 0`, and also on an instance whose `up` stopped arriving after it reported in the last day. - **Why.** On the Ansible path, and with Compose's `alloy` profile, the scraper is @@ -140,6 +142,41 @@ The certificate's expiry is not covered: iroh-relay renews it itself but exports nothing about it. Watch `https:///healthz` with a blackbox or synthetic check of your own. +## iroh DNS server + +`iroh-dns-server/` holds the same pair for the self-hosted iroh DNS server +([`roles/iroh_dns_server`](../ansible/roles/iroh_dns_server/README.md)), the pkarr +relay nodes publish to and the DNS server clients resolve them through. It runs on +the Ansible path and as Compose's `dns` profile. The chart never renders it. + +| File | What it is | +|------|------------| +| `iroh-dns-server/dashboard-iroh-dns-server.json` | `uid: decdn-iroh-dns-server`: up, publishes and DNS requests per day, zones cached, mean HTTP duration, pkarr publishes (update and no-op), the record store, DNS requests by transport, lookups by answer, HTTP requests, the server's log. | +| `iroh-dns-server/prometheus-alerts.yml` | Rule group `iroh-dns-server`: down, lookup errors rising against the server's own last day, publishes stopped (no pkarr put in 30 minutes after some in the last day; nodes republish every 5). | +| `iroh-dns-server/exported-metrics.txt` | The `/metrics` of the pinned `iroh-dns-server` release. `make lint-helm` fails on a `dns_server_*` name the dashboard or rules use that is not in it. | + +They select `job="iroh-dns-server"`, which the `grafana_alloy` role stamps on the +server's `/metrics` (`grafana_alloy_iroh_dns_server_job`); the log panel selects +`unit="iroh-dns-server.service"` with the parsed `level`. On Compose, the `alloy` +profile on the server's host scrapes `127.0.0.1:9117/metrics` as +`job="iroh-dns-server"` and labels the container's journal lines +`unit="iroh-dns-server.service"`, so the dashboard and its log panel work unchanged; +without that profile, the scrape is your own. Import them as above, e.g. +`mimirtool rules load iroh-dns-server/prometheus-alerts.yml`. `make lint-helm` also +runs `promtool check rules` and the unit tests in +`iroh-dns-server/prometheus-alerts_test.yml`. + +The lookup-error rule is a rise, not a share: the server counts every answer other +than NOERROR and NXDOMAIN as an error, including the NOTIMP it gives a scanner's +UPDATE or STATUS probe, so a public udp/53 has a steady floor of them. A query for +a name outside its origins is NXDOMAIN and counts as not found. + +The certificate's expiry is not covered: iroh-dns-server renews it itself but +exports nothing about it. Watch `https:///healthz` with a blackbox or +synthetic check of your own, and the delegation with an external `dig` (e.g. +`dig SOA` through a public resolver). `IrohDnsServerNoPublishes` is the +symptom when nodes cannot reach the https listener. + ## Public gateway `decdn-gateway/` holds the same pair for the public gateway diff --git a/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json new file mode 100644 index 00000000..e0a84322 --- /dev/null +++ b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json @@ -0,0 +1,1073 @@ +{ + "annotations": { + "list": [ + { + "builtIn": 1, + "datasource": { + "type": "grafana", + "uid": "-- Grafana --" + }, + "enable": true, + "hide": true, + "iconColor": "rgba(0, 211, 255, 1)", + "name": "Annotations & Alerts", + "type": "dashboard" + } + ] + }, + "description": "The self-hosted iroh DNS server deCDN nodes publish to and clients resolve through (roles/iroh_dns_server): pkarr publishes, the record store, DNS requests and lookups, HTTP, logs. Metrics job=\"iroh-dns-server\" as the grafana_alloy role ships them.", + "editable": true, + "graphTooltip": 1, + "links": [], + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 1, + "panels": [], + "title": "DNS server", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "iroh-dns-server", + "description": "Whether Prometheus can scrape iroh-dns-server (`up`). Down: nodes cannot publish their records and clients cannot resolve them under its origin. An instant query: with the host or its agent gone it shows no data, never a stale UP.", + "type": "stat", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "up{job=\"iroh-dns-server\", instance=~\"$instance\"}", + "refId": "A", + "legendFormat": "{{instance}}", + "instant": true, + "range": false + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "none", + "mappings": [ + { + "type": "value", + "options": { + "0": { + "text": "DOWN" + }, + "1": { + "text": "UP" + } + } + } + ], + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "textMode": "auto", + "wideLayout": true, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "gridPos": { + "h": 5, + "w": 4, + "x": 0, + "y": 1 + }, + "id": 2 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Publishes, 24h", + "description": "pkarr puts in the last 24 hours, whether they changed the stored record or not (dns_server_pkarr_publish_update_total + dns_server_pkarr_publish_noop_total). Every node republishes every 5 minutes, about 288 a day each.", + "type": "stat", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "increase(dns_server_pkarr_publish_update_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[1d]) + increase(dns_server_pkarr_publish_noop_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[1d])", + "refId": "A", + "legendFormat": "{{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "mappings": [], + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "textMode": "auto", + "wideLayout": true, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "gridPos": { + "h": 5, + "w": 5, + "x": 4, + "y": 1 + }, + "id": 3 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "DNS requests, 24h", + "description": "DNS requests over every transport in the last 24 hours (dns_server_dns_requests_total).", + "type": "stat", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "increase(dns_server_dns_requests_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[1d])", + "refId": "A", + "legendFormat": "{{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "mappings": [], + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "textMode": "auto", + "wideLayout": true, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "gridPos": { + "h": 5, + "w": 5, + "x": 9, + "y": 1 + }, + "id": 4 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Zones cached", + "description": "Zones in the server's in-memory cache (dns_server_cache_zones, an LRU of up to about a million): one per key that published or was looked up since the server started.", + "type": "stat", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "dns_server_cache_zones{job=\"iroh-dns-server\", instance=~\"$instance\"}", + "refId": "A", + "legendFormat": "{{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short", + "mappings": [], + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "textMode": "auto", + "wideLayout": true, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "gridPos": { + "h": 5, + "w": 5, + "x": 14, + "y": 1 + }, + "id": 5 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Mean HTTP duration, 24h", + "description": "Mean duration of the HTTP requests (pkarr puts and gets, DNS-over-HTTPS) in the last 24 hours: dns_server_http_requests_duration_ms_total over dns_server_http_requests_total.", + "type": "stat", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "increase(dns_server_http_requests_duration_ms_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[1d]) / increase(dns_server_http_requests_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[1d])", + "refId": "A", + "legendFormat": "{{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ms", + "mappings": [], + "decimals": 0 + }, + "overrides": [] + }, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "auto", + "orientation": "auto", + "textMode": "auto", + "wideLayout": true, + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + } + }, + "gridPos": { + "h": 5, + "w": 5, + "x": 19, + "y": 1 + }, + "id": 6 + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 6 + }, + "id": 7, + "panels": [], + "title": "pkarr", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Publishes per minute", + "description": "pkarr puts (PUT /pkarr) that changed the stored record (update: a node's addresses moved, or a new node) and that did not (no-op: the 5-minute republish). A flat zero on a server nodes point at fires IrohDnsServerNoPublishes.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_pkarr_publish_update_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "A", + "legendFormat": "update {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_pkarr_publish_noop_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "B", + "legendFormat": "no-op {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 7 + }, + "id": 8 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Record store per minute", + "description": "Signed packets inserted (a key first seen), updated (replaced by a newer one) and expired (evicted 7 days after its last put) in the store. Removed is always 0 in 1.3.0.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_store_packets_inserted_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "A", + "legendFormat": "inserted {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_store_packets_updated_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "B", + "legendFormat": "updated {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_store_packets_expired_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "C", + "legendFormat": "expired {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_store_packets_removed_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "D", + "legendFormat": "removed {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 7 + }, + "id": 9 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Cached zones", + "description": "dns_server_cache_zones (the main cache) and dns_server_cache_zones_dht (zones resolved from the mainline DHT, which stays empty with iroh_dns_server_mainline_enabled off).", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "dns_server_cache_zones{job=\"iroh-dns-server\", instance=~\"$instance\"}", + "refId": "A", + "legendFormat": "main {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "dns_server_cache_zones_dht{job=\"iroh-dns-server\", instance=~\"$instance\"}", + "refId": "B", + "legendFormat": "DHT {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 7 + }, + "id": 10 + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 15 + }, + "id": 11, + "panels": [], + "title": "DNS", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "DNS requests per minute", + "description": "By transport: udp (dns_server_dns_requests_udp_total), DNS-over-HTTPS (dns_server_dns_requests_https_total), and tcp, which iroh-dns-server counts in dns_server_dns_requests_total only, so it is the total less the other two.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_dns_requests_udp_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "A", + "legendFormat": "udp {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_dns_requests_https_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "B", + "legendFormat": "https {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "(rate(dns_server_dns_requests_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60) - (rate(dns_server_dns_requests_udp_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60) - (rate(dns_server_dns_requests_https_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60)", + "refId": "C", + "legendFormat": "tcp {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 12 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Lookups per minute", + "description": "Success: at least one answer. Not found: NXDOMAIN or no answer, including every query for a name outside the origins (the server is authoritative for the root too). Error: any other answer: SERVFAIL (the record store failing) or NOTIMP (a non-query opcode, usually a scanner). A sustained rise fires IrohDnsServerLookupErrors.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_dns_lookup_success_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "A", + "legendFormat": "success {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_dns_lookup_notfound_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "B", + "legendFormat": "not found {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_dns_lookup_error_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "C", + "legendFormat": "error {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 13 + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 24 + }, + "id": 14, + "panels": [], + "title": "HTTP", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "HTTP requests per minute", + "description": "Requests on the https listener (pkarr puts and gets, DNS-over-HTTPS) and the loopback health listener. Error is any non-2xx, so a pkarr get for a key the server does not hold (404) counts: a share of errors is normal.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_http_requests_success_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "A", + "legendFormat": "2xx {{instance}}" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_http_requests_error_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) * 60", + "refId": "B", + "legendFormat": "non-2xx {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "short", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 25 + }, + "id": 15 + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "title": "Mean HTTP duration", + "description": "dns_server_http_requests_duration_ms_total over dns_server_http_requests_total: the mean only, iroh-dns-server exports no histogram.", + "type": "timeseries", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "rate(dns_server_http_requests_duration_ms_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval]) / rate(dns_server_http_requests_total{job=\"iroh-dns-server\", instance=~\"$instance\"}[$__rate_interval])", + "refId": "A", + "legendFormat": "mean {{instance}}" + } + ], + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": false, + "stacking": { + "group": "A", + "mode": "none" + } + }, + "unit": "ms", + "min": 0 + }, + "overrides": [] + }, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 25 + }, + "id": 16 + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 33 + }, + "id": 17, + "panels": [], + "title": "Logs", + "type": "row" + }, + { + "datasource": { + "type": "loki", + "uid": "${DS_LOKI}" + }, + "title": "iroh-dns-server log", + "description": "The server's journal (unit iroh-dns-server.service), level parsed by the grafana_alloy role. Info by default (iroh_dns_server_log_level).", + "type": "logs", + "targets": [ + { + "datasource": { + "type": "loki", + "uid": "${DS_LOKI}" + }, + "refId": "A", + "queryType": "range", + "expr": "{unit=\"iroh-dns-server.service\", instance=~\"$instance\", level=~\"$level\"}" + } + ], + "options": { + "dedupStrategy": "none", + "enableInfiniteScrolling": true, + "enableLogDetails": true, + "prettifyLogMessage": false, + "showCommonLabels": false, + "showLabels": false, + "showTime": true, + "sortOrder": "Descending", + "wrapLogMessage": true + }, + "fieldConfig": { + "defaults": {}, + "overrides": [] + }, + "gridPos": { + "h": 10, + "w": 24, + "x": 0, + "y": 34 + }, + "id": 18 + } + ], + "refresh": "30s", + "schemaVersion": 39, + "tags": [ + "decdn", + "iroh-dns-server" + ], + "templating": { + "list": [ + { + "current": { + "selected": false, + "text": "grafanacloud-decdn-prom", + "value": "grafanacloud-prom" + }, + "hide": 0, + "includeAll": false, + "label": "Metrics", + "multi": false, + "name": "DS_PROMETHEUS", + "options": [], + "query": "prometheus", + "queryValue": "", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "current": { + "selected": false, + "text": "grafanacloud-decdn-logs", + "value": "grafanacloud-logs" + }, + "hide": 0, + "includeAll": false, + "label": "Logs", + "multi": false, + "name": "DS_LOKI", + "options": [], + "query": "loki", + "queryValue": "", + "refresh": 1, + "regex": "", + "skipUrlSync": false, + "type": "datasource" + }, + { + "allValue": ".*", + "current": { + "selected": true, + "text": [ + "All" + ], + "value": [ + "$__all" + ] + }, + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "definition": "label_values(up{job=\"iroh-dns-server\"}, instance)", + "hide": 0, + "includeAll": true, + "label": "Instance", + "multi": true, + "name": "instance", + "options": [], + "query": { + "qryType": 1, + "query": "label_values(up{job=\"iroh-dns-server\"}, instance)", + "refId": "PrometheusVariableQueryEditor-VariableQuery" + }, + "refresh": 2, + "regex": "", + "skipUrlSync": false, + "sort": 1, + "type": "query" + }, + { + "allValue": ".*", + "current": { + "selected": true, + "text": [ + "All" + ], + "value": [ + "$__all" + ] + }, + "hide": 0, + "includeAll": true, + "label": "Log level", + "multi": true, + "name": "level", + "options": [], + "query": "debug,info,warning,error", + "skipUrlSync": false, + "type": "custom" + } + ] + }, + "time": { + "from": "now-24h", + "to": "now" + }, + "timepicker": {}, + "timezone": "", + "title": "deCDN — iroh DNS server", + "uid": "decdn-iroh-dns-server", + "version": 1 +} diff --git a/monitoring/iroh-dns-server/exported-metrics.txt b/monitoring/iroh-dns-server/exported-metrics.txt new file mode 100644 index 00000000..fec05400 --- /dev/null +++ b/monitoring/iroh-dns-server/exported-metrics.txt @@ -0,0 +1,63 @@ +# The /metrics exposition of iroh-dns-server 1.3.0 (n0-computer/iroh, x86_64-unknown-linux-gnu +# release asset), captured from a fresh process with no traffic, so every value is 0. +# Every dns_server_* name a dashboard or rule in this directory uses must appear +# here (`make lint-helm` checks). Re-capture on an iroh_dns_server_version bump, with +# a config like the role's on unprivileged ports ([dns] port, [http] port, no [https], +# [mainline] enabled = false) and [metrics] bind_addr = "127.0.0.1:9117": +# iroh-dns-server --config +# curl -s http://127.0.0.1:9117/metrics +# HELP dns_server_pkarr_publish_update Number of pkarr relay puts that updated the stored packet. +# TYPE dns_server_pkarr_publish_update counter +dns_server_pkarr_publish_update_total 0 +# HELP dns_server_pkarr_publish_noop Number of pkarr relay puts that did not change the stored packet. +# TYPE dns_server_pkarr_publish_noop counter +dns_server_pkarr_publish_noop_total 0 +# HELP dns_server_dns_requests Total number of DNS requests across all transports. +# TYPE dns_server_dns_requests counter +dns_server_dns_requests_total 0 +# HELP dns_server_dns_requests_udp Number of DNS requests received over UDP. +# TYPE dns_server_dns_requests_udp counter +dns_server_dns_requests_udp_total 0 +# HELP dns_server_dns_requests_https Number of DNS requests received over HTTPS (DoH). +# TYPE dns_server_dns_requests_https counter +dns_server_dns_requests_https_total 0 +# HELP dns_server_dns_lookup_success Number of DNS lookups that returned at least one answer. +# TYPE dns_server_dns_lookup_success counter +dns_server_dns_lookup_success_total 0 +# HELP dns_server_dns_lookup_notfound Number of DNS lookups that returned no answers. +# TYPE dns_server_dns_lookup_notfound counter +dns_server_dns_lookup_notfound_total 0 +# HELP dns_server_dns_lookup_error Number of DNS lookups that failed with an error. +# TYPE dns_server_dns_lookup_error counter +dns_server_dns_lookup_error_total 0 +# HELP dns_server_http_requests Number of HTTP requests served. +# TYPE dns_server_http_requests counter +dns_server_http_requests_total 0 +# HELP dns_server_http_requests_success Number of HTTP requests that returned a 2xx status code. +# TYPE dns_server_http_requests_success counter +dns_server_http_requests_success_total 0 +# HELP dns_server_http_requests_error Number of HTTP requests that returned a non-2xx status code. +# TYPE dns_server_http_requests_error counter +dns_server_http_requests_error_total 0 +# HELP dns_server_http_requests_duration_ms Cumulative duration of all HTTP requests, in milliseconds. +# TYPE dns_server_http_requests_duration_ms counter +dns_server_http_requests_duration_ms_total 0 +# HELP dns_server_store_packets_inserted Number of signed packets newly inserted into the store. +# TYPE dns_server_store_packets_inserted counter +dns_server_store_packets_inserted_total 0 +# HELP dns_server_store_packets_removed Number of signed packets removed from the store. +# TYPE dns_server_store_packets_removed counter +dns_server_store_packets_removed_total 0 +# HELP dns_server_store_packets_updated Number of times an existing signed packet was replaced by a newer one. +# TYPE dns_server_store_packets_updated counter +dns_server_store_packets_updated_total 0 +# HELP dns_server_store_packets_expired Number of signed packets removed by the eviction task. +# TYPE dns_server_store_packets_expired counter +dns_server_store_packets_expired_total 0 +# HELP dns_server_cache_zones Current number of zones in the main cache. +# TYPE dns_server_cache_zones gauge +dns_server_cache_zones 0 +# HELP dns_server_cache_zones_dht Current number of zones in the DHT cache. +# TYPE dns_server_cache_zones_dht gauge +dns_server_cache_zones_dht 0 +# EOF diff --git a/monitoring/iroh-dns-server/prometheus-alerts.yml b/monitoring/iroh-dns-server/prometheus-alerts.yml new file mode 100644 index 00000000..3e7d5679 --- /dev/null +++ b/monitoring/iroh-dns-server/prometheus-alerts.yml @@ -0,0 +1,129 @@ +# iroh-dns-server reference Prometheus alerting rules +# +# For the self-hosted iroh DNS server deCDN nodes publish their records to and +# clients resolve them through (roles/iroh_dns_server), whose /metrics the +# grafana_alloy role (and Compose's alloy profile) ships as job="iroh-dns-server". +# Every series is in iroh-dns-server's `dns_server` group; the names are the ones +# the pinned release exports (exported-metrics.txt beside this file, which +# `make lint-helm` checks them against). Thresholds are operator-onboarding +# defaults, not normative. +# +# NOT covered: the certificate's expiry. iroh-dns-server renews it itself (Let's +# Encrypt) but exports nothing about it; watch https:///healthz with a +# blackbox or synthetic check of your own, and the DNS side with an external dig +# (dig @ SOA) from outside the host. IrohDnsServerNoPublishes +# is the symptom when nodes cannot reach the https listener. +# +# The server runs on the Ansible path and as Compose's `dns` profile. The +# decdn-node chart does NOT render this file (its PrometheusRule reads +# ../decdn-node/prometheus-alerts.yml). +# +# Validate locally with `make lint-helm`: it runs promtool check rules on this +# file, and promtool test rules on prometheus-alerts_test.yml against a copy +# with the annotations stripped (see that file). +# +# Load into Grafana Cloud with mimirtool (see ../README.md), or into Prometheus via +# rule_files (example in ../decdn-node/prometheus-alerts.yml). + +groups: + - name: iroh-dns-server + interval: 30s + rules: + # The second arm fires on a series that stopped arriving. An agent scraping on the + # host (grafana_alloy) writes no up=0 when the host, its network or the agent itself + # is gone, so `up == 0` alone stays silent when the whole host is. (A central + # Prometheus writes up=0 itself; there the arm fires once the target leaves service + # discovery.) The arm keeps the instance label (absent() would not). Both arms + # reduce to job and instance, the labels the alert carries: a changed region or + # other label on a live target does not fire, one target fires once even with + # two label sets in the window, and the for: carries over between the arms. It + # remembers a series for 1d, so it resolves a day after the last sample even if + # the host is still dead, and a decommissioned host or a renamed job or instance + # fires until then: silence it. + - alert: IrohDnsServerDown + expr: | + max by (job, instance) (up{job="iroh-dns-server"} == 0) + or max by (job, instance) ( + max_over_time(up{job="iroh-dns-server"}[1d]) unless on(job, instance) up{job="iroh-dns-server"}) + for: 2m + labels: + component: dns + severity: critical + annotations: + summary: "iroh-dns-server on {{ $labels.instance }} unreachable" + description: | + iroh-dns-server's scrape has failed for two minutes, or its series + stopped arriving (the host, its network or the scraping agent is down). + Nodes cannot publish their records and clients cannot resolve + them under its origin (decdn_discovery_dns_origin); upstream has + no replication, so there is no second server to fall back to. + Read why on the host: journalctl -u iroh-dns-server -e. Adjust + the job="iroh-dns-server" selector if your scrape job is named + differently. + + # A rise against the server's own last day, not a share of lookups. In 1.3.0 + # dns_lookup_error counts every answer other than NOERROR and NXDOMAIN: a + # SERVFAIL (the record store failing) and a NOTIMP (a non-query opcode, such + # as an UPDATE or STATUS probe from a scanner). A query for a name outside + # the origins is NXDOMAIN (the server is authoritative for "." too), so it + # counts as not-found, and a packet that does not parse is not counted. The + # probes on a public udp/53 are steady, so the baseline absorbs them, and a + # quiet server could otherwise sit above any fixed share. The floor (0.1/s, + # 6 a minute) keeps a server with no baseline from firing on a few probes. + # A server less than a day old has an understated baseline (rate() divides + # by the whole day), so it fires sooner. + - alert: IrohDnsServerLookupErrors + expr: >- + sum by (job, instance) (rate(dns_server_dns_lookup_error_total{job="iroh-dns-server"}[30m])) > 0.1 + and on(job, instance) + sum by (job, instance) (rate(dns_server_dns_lookup_error_total{job="iroh-dns-server"}[30m])) + > 3 * sum by (job, instance) (rate(dns_server_dns_lookup_error_total{job="iroh-dns-server"}[1d])) + for: 15m + labels: + component: dns + severity: warning + annotations: + summary: "iroh-dns-server on {{ $labels.instance }} answers more lookups with an error" + description: | + iroh-dns-server answered {{ $value | humanize }} lookups a second + with an error (neither NOERROR nor NXDOMAIN) over the last 30 + minutes, more than three times its rate over the last day, for + 15 minutes. SERVFAIL means the record store is failing: look for + ERROR lines in journalctl -u iroh-dns-server and check the + disk under /var/lib/iroh-dns-server. NOTIMP means someone sends + non-query opcodes (an UPDATE or STATUS scan); harmless, and + absorbed into the baseline once it is steady. + + # Every node republishes its record every 5 minutes, so 30 minutes is six + # missed rounds from every node. The second arm keeps it to a server that + # had puts in the last day: one no node points at (or a fresh one) stays + # silent, and it resolves a day after the last put even if none came back. + - alert: IrohDnsServerNoPublishes + expr: >- + ( + sum by (job, instance) (increase(dns_server_pkarr_publish_update_total{job="iroh-dns-server"}[30m])) + + sum by (job, instance) (increase(dns_server_pkarr_publish_noop_total{job="iroh-dns-server"}[30m])) + ) == 0 + and on(job, instance) + ( + sum by (job, instance) (increase(dns_server_pkarr_publish_update_total{job="iroh-dns-server"}[1d])) + + sum by (job, instance) (increase(dns_server_pkarr_publish_noop_total{job="iroh-dns-server"}[1d])) + ) > 0 + for: 5m + labels: + component: dns + severity: warning + annotations: + summary: "No node has published to iroh-dns-server on {{ $labels.instance }} for 35 minutes" + description: | + iroh-dns-server took no pkarr put (PUT /pkarr) in the last 30 + minutes, for 5 minutes, after taking some in the last day. + Nodes republish every 5 minutes, so they cannot reach it, or + stopped pointing at it. Check the https listener and its + certificate (curl -v https:///healthz from outside), + the nodes' decdn_discovery_pkarr_url and + decdn_discovery_dns_origin, and whether + iroh_dns_server_pkarr_put_rate_limit refuses them. Stored + records keep resolving until they expire, 7 days after their + last put, but a node whose addresses change meanwhile is + resolved to the old ones. diff --git a/monitoring/iroh-dns-server/prometheus-alerts_test.yml b/monitoring/iroh-dns-server/prometheus-alerts_test.yml new file mode 100644 index 00000000..9510647f --- /dev/null +++ b/monitoring/iroh-dns-server/prometheus-alerts_test.yml @@ -0,0 +1,159 @@ +# promtool unit tests for prometheus-alerts.yml beside it, run by the chart's +# tests/render-test.sh (`make lint-helm`). The rules compare a counter's rate with +# its own over a day, and sum two counters' increases, which promtool check cannot +# catch. The test clock starts at 0. +# +# It does not run in place: render-test.sh copies it into a scratch directory as +# iroh-dns-server-alerts_test.yml, beside a copy of prometheus-alerts.yml with the +# annotations stripped (promtool compares annotations exactly, and they are prose) +# named iroh-dns-server-alerts.yml, the name rule_files below expects. +rule_files: + - iroh-dns-server-alerts.yml + +evaluation_interval: 30s + +tests: + # Down. a fails its scrape (up=0) and fires at exactly the for, under job and + # instance alone (its region label is dropped). Its samples end at 5m too, and + # the absence arm carries the alert on without resetting the for (at 10m30s a + # for restarted at the handoff would not have run out yet). b is + # healthy throughout. c stops at 5m with no staleness marker (the host died with + # its agent): up goes absent once the 5m lookback passes, the absence arm fires + # after the for, is still firing at 23h and resolves once the 1d window holds no + # sample (25h). d gets a staleness marker and is back 1m later, inside the for, + # so it never fires (checked at 6m, inside the gap). d2's marker is followed by + # a gap longer than the for: it fires at 7m30s and resolves when it is back + # (10m30s). e changes its region label on a live target and does not fire; + # when it stops later (15m30s) it fires once, under job and instance alone, + # although both of its label sets are still in the 1d window. + - interval: 30s + input_series: + - series: 'up{job="iroh-dns-server", instance="a", region="eu"}' + values: '0x10' + - series: 'up{job="iroh-dns-server", instance="b"}' + values: '1x3000' + - series: 'up{job="iroh-dns-server", instance="c"}' + values: '1x10' + - series: 'up{job="iroh-dns-server", instance="d"}' + values: '1x10 stale _x1 1x2987' + - series: 'up{job="iroh-dns-server", instance="d2"}' + values: '1x10 stale _x9 1x2979' + - series: 'up{job="iroh-dns-server", instance="e", region="eu"}' + values: '1x10' + - series: 'up{job="iroh-dns-server", instance="e", region="us"}' + values: '_x11 1x20' + alert_rule_test: + - eval_time: 1m30s + alertname: IrohDnsServerDown + exp_alerts: [] + - eval_time: 2m + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - eval_time: 6m + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - eval_time: 8m + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - exp_labels: {job: iroh-dns-server, instance: d2, component: dns, severity: critical} + - eval_time: 10m30s + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - eval_time: 13m + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - exp_labels: {job: iroh-dns-server, instance: c, component: dns, severity: critical} + - eval_time: 23h + alertname: IrohDnsServerDown + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: critical} + - exp_labels: {job: iroh-dns-server, instance: c, component: dns, severity: critical} + - exp_labels: {job: iroh-dns-server, instance: e, component: dns, severity: critical} + - eval_time: 25h + alertname: IrohDnsServerDown + exp_alerts: [] + + + # Lookup errors, per minute. a answers 3 errors a minute (0.05/s, under the + # floor) for 25h, then 60 (1/s) from 1501m: it fires once the rise has lasted + # the for, not before, and keeps firing while the day's rate is still low. b + # answers 60 a minute from the start: steady, so it never fires once it has a + # day behind it, but in its first hours its 1d rate is understated (the + # documented young-server case) and it fires. c has no errors for 25h, then 9 + # a minute (0.15/s): over the floor with no baseline, so it fires, later than a + # because its 30m rate takes longer to pass the floor. d has no errors, then 3 + # a minute: a rise, but under the floor, so it never fires. + - interval: 1m + input_series: + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="a"}' + values: '0+3x1500 4560+60x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="b"}' + values: '0+60x1650' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="c"}' + values: '0x1500 9+9x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="d"}' + values: '0x1500 3+3x150' + alert_rule_test: + - eval_time: 1h + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: b, component: dns, severity: warning} + - eval_time: 1500m + alertname: IrohDnsServerLookupErrors + exp_alerts: [] + - eval_time: 1510m + alertname: IrohDnsServerLookupErrors + exp_alerts: [] + - eval_time: 1530m + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} + - eval_time: 1560m + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: c, component: dns, severity: warning} + + # Publishes, per minute. a takes updates and no-op puts until 60m, then none: no + # put in 30m from 90m, so it fires after the for (95m), keeps firing while the + # last put is in the 1d window, and resolves once it is not (25h). b never took + # a put, so it never fires. c takes no-op puts only, which count. d restarts at + # 40m (its counters reset) and goes on taking puts: no alert. + - interval: 1m + input_series: + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="a"}' + values: '0+2x60 120x1600' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="a"}' + values: '0+10x60 600x1600' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="b"}' + values: '0x1660' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="b"}' + values: '0x1660' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="c"}' + values: '0x1660' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="c"}' + values: '0+5x1660' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="d"}' + values: '0+2x40 0+2x1620' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="d"}' + values: '0+10x40 0+10x1620' + alert_rule_test: + - eval_time: 92m + alertname: IrohDnsServerNoPublishes + exp_alerts: [] + - eval_time: 96m + alertname: IrohDnsServerNoPublishes + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} + - eval_time: 23h + alertname: IrohDnsServerNoPublishes + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} + - eval_time: 25h30m + alertname: IrohDnsServerNoPublishes + exp_alerts: [] From de2a83c6170361cc105161c0de62a2fde239ff1b Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sun, 1 Nov 2026 15:55:17 +0300 Subject: [PATCH 2/3] fix(monitoring): correct iroh-dns-server semantics and pin the rules' boundaries Review of #201 against upstream v1.3.0 (and the real binary): - every node republish is a pkarr *update* (re-signed with a fresh timestamp); no-op is only a put older than the stored packet. The publishes and record-store panels said the opposite. - a failing record store answers REFUSED (node_zone_handler's err_refused), not SERVFAIL; the rule comment, annotation (now naming the store's real WARN/ERROR lines), the lookups panel and the README say so. An AXFR is NOERROR with no answer (checked on the binary), so it counts as not found. - dns_server_cache_zones is the lookup cache: a key enters on a lookup and leaves on its next publish. It is not a node count. - the mean HTTP duration includes /healthz and truncates to whole ms. - a series under ~8 hours old fires on any steady error rate over the floor; say so in the rule and its annotation. Tests: pin the 3x multiplier (2.5x silent, 4x fires), the 0.1/s floor (0.083 vs 0.117), the 15m for to the minute, the 1d memory and resolution of LookupErrors; each summed publish counter on its own in both arms, resolution when puts resume, the reset inside a window, and the 94m fire time. Also: the root README's monitoring row, NoPublishes silencing on Compose, both dig checks named alike, AGENTS.md no longer implies test-scripts checks exported-metrics.txt, and render-test cleans up its new negative. Co-Authored-By: Claude Opus 5.5 --- AGENTS.md | 5 +- README.md | 2 +- ansible/roles/iroh_dns_server/README.md | 6 +- charts/decdn-node/tests/render-test.sh | 4 +- monitoring/README.md | 17 ++- .../dashboard-iroh-dns-server.json | 14 +-- .../iroh-dns-server/prometheus-alerts.yml | 40 +++--- .../prometheus-alerts_test.yml | 116 +++++++++++++++--- 8 files changed, 151 insertions(+), 53 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index db7ace44..23b5a914 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -228,8 +228,9 @@ boundaries or exists nowhere else. stub is the holder; the role never touches the host's resolver. - A version bump (`iroh_relay_version`, `iroh_dns_server_version`, both tracking the iroh decdn builds against) moves every per-target sha256, the image digest and - version comment in `compose/compose.yaml`, `DNS_VERSION` in `decdn-compose` and each - one's `exported-metrics.txt` (`make test-scripts` checks the names agree). The + version comment in `compose/compose.yaml` and `DNS_VERSION` in `decdn-compose` + (`make test-scripts` checks they agree with the role pins), and each one's + `exported-metrics.txt` (re-captured by hand; nothing checks its version header). The relay's allowlist play reads `decdn whoami` on every `decdn_nodes` host by `delegate_to`, so `LIMIT=` still needs those hosts reachable; relays deploy `serial: 1`. diff --git a/README.md b/README.md index c2c0c950..a4d0cf46 100644 --- a/README.md +++ b/README.md @@ -167,7 +167,7 @@ This is the canonical statement; the per-path READMEs add only what is specific | [`cloud-init/`](cloud-init/README.md) | The cloud-init deploy path: `user-data-node.yaml` (node operators: a cache node), `user-data-publisher.yaml` (publishers: an origin node, sponsord and its onramp), the on-host `bootstrap.sh`, and the pinned ansible-core and collections it installs. | | [`compose/`](compose/README.md) | The Docker Compose deploy path: the node, `sponsord`, `sponsord-onramp`, an iroh relay, an iroh DNS server, a public gateway and Grafana Alloy, one profile each, and the `decdn-compose` wrapper. | | [`charts/decdn-node/`](charts/decdn-node/README.md) | The Helm chart; it renders the node's dashboards and alert rules from `monitoring/`. | -| [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose) and the iroh relay (`iroh-relay/`, Ansible and Compose). | +| [`monitoring/`](monitoring/README.md) | The deCDN Grafana dashboards and Prometheus alert rules, for the node (`decdn-node/`, every deploy path), sponsord (`sponsord/`, Ansible and Compose), the iroh relay (`iroh-relay/`, Ansible and Compose), the iroh DNS server (`iroh-dns-server/`, Ansible and Compose) and the public gateway (`decdn-gateway/`, Ansible). | | [`docs/`](docs/requirements.md) | Cross-path operator docs: the two front doors ([node operators](docs/node-operators.md), [publishers](docs/publishers.md)), requirements, lifecycle. | | [`examples/collection-fleet/`](examples/collection-fleet/README.md) | A starting point for a private repository that deploys from the Galaxy collections alone, with only its own inventory. | | `scripts/` | Generators for the upstream mirrors (network profiles) and Compose's Alloy config (`render-compose-alloy.sh`), and the release gate. | diff --git a/ansible/roles/iroh_dns_server/README.md b/ansible/roles/iroh_dns_server/README.md index f5d54030..46497a27 100644 --- a/ansible/roles/iroh_dns_server/README.md +++ b/ansible/roles/iroh_dns_server/README.md @@ -226,8 +226,10 @@ then: The dashboard and alert rules (down, lookup errors rising, publishes stopped) are in [`monitoring/iroh-dns-server/`](../../../monitoring/README.md#iroh-dns-server). -Certificate expiry is not exported. Watch `https:///healthz` and an -external `dig` with a check of your own. +Certificate expiry is not exported. Watch `https:///healthz`, and DNS +with an external `dig` (the server itself and the delegation, as +[`monitoring/README.md`](../../../monitoring/README.md#iroh-dns-server) shows), with +a check of your own. ## Day 2 diff --git a/charts/decdn-node/tests/render-test.sh b/charts/decdn-node/tests/render-test.sh index 14073574..2becb235 100755 --- a/charts/decdn-node/tests/render-test.sh +++ b/charts/decdn-node/tests/render-test.sh @@ -63,7 +63,8 @@ pass "schema-key checker: good/bad/empty fixtures" # operators on every deploy path import from; the chart reaches them through # files/monitoring, a relative symlink, because .Files cannot read outside the chart. # helm package must turn it into regular files, and nothing of sponsord's, -# iroh-relay's, iroh-dns-server's or decdn-gateway's may ship, nor artifacthub-repo.yml (release-chart.yml pushes it on its own). +# iroh-relay's, iroh-dns-server's or decdn-gateway's may ship, nor +# artifacthub-repo.yml (release-chart.yml pushes it on its own). link_target="../../../monitoring/decdn-node" # Prints what is wrong and returns non-zero, so the negatives below can call it too. check_monitoring_link() { # @@ -147,6 +148,7 @@ expect_link_fail "the decdn-gateway dashboard copied into monitoring/decdn-node" rm "$neg/monitoring/decdn-node/dashboard-decdn-gateway.json" cp "$neg/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json" "$neg/monitoring/decdn-node/" expect_link_fail "the iroh-dns-server dashboard copied into monitoring/decdn-node" "sponsord, iroh-relay, iroh-dns-server or decdn-gateway files in the chart package" +rm "$neg/monitoring/decdn-node/dashboard-iroh-dns-server.json" # --- positive renders ----------------------------------------------------------- for values in "$chart"/ci/*.yaml; do diff --git a/monitoring/README.md b/monitoring/README.md index 5e5179c3..055611bc 100644 --- a/monitoring/README.md +++ b/monitoring/README.md @@ -168,13 +168,22 @@ runs `promtool check rules` and the unit tests in The lookup-error rule is a rise, not a share: the server counts every answer other than NOERROR and NXDOMAIN as an error, including the NOTIMP it gives a scanner's -UPDATE or STATUS probe, so a public udp/53 has a steady floor of them. A query for -a name outside its origins is NXDOMAIN and counts as not found. +UPDATE or STATUS probe, so a public udp/53 has a steady floor of them. A failing +record store answers REFUSED, which counts too. A query for a name outside its +origins is NXDOMAIN and an AXFR is NOERROR with no answer, so both count as not +found. In a server's first ~8 hours its day's baseline is short, and any steady +error rate over the floor fires. + +`IrohDnsServerNoPublishes`, like the `*Down` alerts, fires for up to a day on a +server that went quiet on purpose: silence it before you re-point the nodes +elsewhere, decommission the host, or drop `dns` from a Compose host's +`COMPOSE_PROFILES` ([docs/lifecycle.md](../docs/lifecycle.md#iroh-dns-server)). The certificate's expiry is not covered: iroh-dns-server renews it itself but exports nothing about it. Watch `https:///healthz` with a blackbox or -synthetic check of your own, and the delegation with an external `dig` (e.g. -`dig SOA` through a public resolver). `IrohDnsServerNoPublishes` is the +synthetic check of your own, and DNS from outside the host with `dig`: the server +itself (`dig @ SOA`) and the delegation through a public +resolver (`dig SOA`). `IrohDnsServerNoPublishes` is the symptom when nodes cannot reach the https listener. ## Public gateway diff --git a/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json index e0a84322..d3a85513 100644 --- a/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json +++ b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json @@ -243,7 +243,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Zones cached", - "description": "Zones in the server's in-memory cache (dns_server_cache_zones, an LRU of up to about a million): one per key that published or was looked up since the server started.", + "description": "Zones in the server's in-memory lookup cache (dns_server_cache_zones, an LRU of up to about a million). A key enters on a DNS lookup and leaves on its next publish, so with nodes republishing every 5 minutes this counts keys resolved in the last few minutes, not nodes.", "type": "stat", "targets": [ { @@ -305,7 +305,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Mean HTTP duration, 24h", - "description": "Mean duration of the HTTP requests (pkarr puts and gets, DNS-over-HTTPS) in the last 24 hours: dns_server_http_requests_duration_ms_total over dns_server_http_requests_total.", + "description": "Mean duration of the HTTP requests (pkarr puts and gets, DNS-over-HTTPS, and the loopback /healthz) in the last 24 hours: dns_server_http_requests_duration_ms_total over dns_server_http_requests_total. Each request counts in whole milliseconds, rounded down, so the mean reads low.", "type": "stat", "targets": [ { @@ -380,7 +380,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Publishes per minute", - "description": "pkarr puts (PUT /pkarr) that changed the stored record (update: a node's addresses moved, or a new node) and that did not (no-op: the 5-minute republish). A flat zero on a server nodes point at fires IrohDnsServerNoPublishes.", + "description": "pkarr puts (PUT /pkarr). Update: every accepted put, including each node's 5-minute republish (re-signed with a fresh timestamp), so a steady fleet shows about one per node every 5 minutes. No-op: a put older than the stored record (a retry overtaken by a newer publish, or a node clock that went back), normally about zero. Both flat on a server nodes point at fires IrohDnsServerNoPublishes.", "type": "timeseries", "targets": [ { @@ -448,7 +448,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Record store per minute", - "description": "Signed packets inserted (a key first seen), updated (replaced by a newer one) and expired (evicted 7 days after its last put) in the store. Removed is always 0 in 1.3.0.", + "description": "Signed packets inserted (a key first seen), updated (replaced by a newer one: every republish, so it tracks the update rate of the panel beside it) and expired (evicted 7 days after its last put) in the store. Removed is always 0 in 1.3.0.", "type": "timeseries", "targets": [ { @@ -534,7 +534,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Cached zones", - "description": "dns_server_cache_zones (the main cache) and dns_server_cache_zones_dht (zones resolved from the mainline DHT, which stays empty with iroh_dns_server_mainline_enabled off).", + "description": "dns_server_cache_zones (the lookup cache: keys resolved since their last publish) and dns_server_cache_zones_dht (zones resolved from the mainline DHT, empty with [mainline] off, as the role and Compose configure it by default).", "type": "timeseries", "targets": [ { @@ -692,7 +692,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Lookups per minute", - "description": "Success: at least one answer. Not found: NXDOMAIN or no answer, including every query for a name outside the origins (the server is authoritative for the root too). Error: any other answer: SERVFAIL (the record store failing) or NOTIMP (a non-query opcode, usually a scanner). A sustained rise fires IrohDnsServerLookupErrors.", + "description": "Success: at least one answer. Not found: NXDOMAIN or no answer, including every query for a name outside the origins (the server is authoritative for the root too). Error: any other answer: REFUSED (the record store failing) or NOTIMP (a non-query opcode, usually a scanner). An AXFR is NOERROR with no answer, so not found. A sustained rise fires IrohDnsServerLookupErrors.", "type": "timeseries", "targets": [ { @@ -850,7 +850,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Mean HTTP duration", - "description": "dns_server_http_requests_duration_ms_total over dns_server_http_requests_total: the mean only, iroh-dns-server exports no histogram.", + "description": "dns_server_http_requests_duration_ms_total over dns_server_http_requests_total: the mean only, iroh-dns-server exports no histogram. Each request counts in whole milliseconds, rounded down, so sub-millisecond requests add 0 and the mean reads low.", "type": "timeseries", "targets": [ { diff --git a/monitoring/iroh-dns-server/prometheus-alerts.yml b/monitoring/iroh-dns-server/prometheus-alerts.yml index 3e7d5679..6012a5a7 100644 --- a/monitoring/iroh-dns-server/prometheus-alerts.yml +++ b/monitoring/iroh-dns-server/prometheus-alerts.yml @@ -10,9 +10,10 @@ # # NOT covered: the certificate's expiry. iroh-dns-server renews it itself (Let's # Encrypt) but exports nothing about it; watch https:///healthz with a -# blackbox or synthetic check of your own, and the DNS side with an external dig -# (dig @ SOA) from outside the host. IrohDnsServerNoPublishes -# is the symptom when nodes cannot reach the https listener. +# blackbox or synthetic check of your own, and DNS from outside the host with dig: +# the server itself (dig @ SOA) and the delegation through a +# public resolver (dig SOA). IrohDnsServerNoPublishes is the symptom +# when nodes cannot reach the https listener. # # The server runs on the Ansible path and as Compose's `dns` profile. The # decdn-node chart does NOT render this file (its PrometheusRule reads @@ -63,15 +64,17 @@ groups: # A rise against the server's own last day, not a share of lookups. In 1.3.0 # dns_lookup_error counts every answer other than NOERROR and NXDOMAIN: a - # SERVFAIL (the record store failing) and a NOTIMP (a non-query opcode, such - # as an UPDATE or STATUS probe from a scanner). A query for a name outside - # the origins is NXDOMAIN (the server is authoritative for "." too), so it - # counts as not-found, and a packet that does not parse is not counted. The - # probes on a public udp/53 are steady, so the baseline absorbs them, and a - # quiet server could otherwise sit above any fixed share. The floor (0.1/s, - # 6 a minute) keeps a server with no baseline from firing on a few probes. - # A server less than a day old has an understated baseline (rate() divides - # by the whole day), so it fires sooner. + # REFUSED (the record store failing: node_zone_handler maps a store error to + # REFUSED, not SERVFAIL) and a NOTIMP (a non-query opcode, such as an UPDATE + # or STATUS probe from a scanner). A query for a name outside the origins is + # NXDOMAIN (the server is authoritative for "." too) and an AXFR is NOERROR + # with no answer, so both count as not-found, and a packet that does not + # parse is not counted. The probes on a public udp/53 are steady, so the + # baseline absorbs them, and a quiet server could otherwise sit above any + # fixed share. The floor (0.1/s, 6 a minute) keeps a server with no baseline + # from firing on a few probes. A series under a day old has an understated + # baseline (rate() divides by the whole day): in its first ~8 hours any + # steady error rate over the floor fires. - alert: IrohDnsServerLookupErrors expr: >- sum by (job, instance) (rate(dns_server_dns_lookup_error_total{job="iroh-dns-server"}[30m])) > 0.1 @@ -88,11 +91,14 @@ groups: iroh-dns-server answered {{ $value | humanize }} lookups a second with an error (neither NOERROR nor NXDOMAIN) over the last 30 minutes, more than three times its rate over the last day, for - 15 minutes. SERVFAIL means the record store is failing: look for - ERROR lines in journalctl -u iroh-dns-server and check the - disk under /var/lib/iroh-dns-server. NOTIMP means someone sends - non-query opcodes (an UPDATE or STATUS scan); harmless, and - absorbed into the baseline once it is steady. + 15 minutes. The record store failing answers REFUSED: look for + "get ... failed" (WARN) or "packet store actor failed" (ERROR) + in journalctl -u iroh-dns-server and check the disk under + /var/lib/iroh-dns-server. NOTIMP means + someone sends non-query opcodes (an UPDATE or STATUS scan); + harmless, and absorbed into the baseline once it is steady. A + server started in the last ~8 hours fires on any steady rate + over the floor: its day's baseline is still short. # Every node republishes its record every 5 minutes, so 30 minutes is six # missed rounds from every node. The second arm keeps it to a server that diff --git a/monitoring/iroh-dns-server/prometheus-alerts_test.yml b/monitoring/iroh-dns-server/prometheus-alerts_test.yml index 9510647f..b5e64cb5 100644 --- a/monitoring/iroh-dns-server/prometheus-alerts_test.yml +++ b/monitoring/iroh-dns-server/prometheus-alerts_test.yml @@ -78,16 +78,15 @@ tests: alertname: IrohDnsServerDown exp_alerts: [] - # Lookup errors, per minute. a answers 3 errors a minute (0.05/s, under the - # floor) for 25h, then 60 (1/s) from 1501m: it fires once the rise has lasted - # the for, not before, and keeps firing while the day's rate is still low. b - # answers 60 a minute from the start: steady, so it never fires once it has a - # day behind it, but in its first hours its 1d rate is understated (the - # documented young-server case) and it fires. c has no errors for 25h, then 9 - # a minute (0.15/s): over the floor with no baseline, so it fires, later than a - # because its 30m rate takes longer to pass the floor. d has no errors, then 3 - # a minute: a rise, but under the floor, so it never fires. + # floor) for 25h, then 60 (1/s) from 1501m: both arms hold from 1504m, so it is + # pending at 1518m and fires at 1519m (the for, to the minute). b answers 60 a + # minute from the start: steady, so it never fires once it has a day behind it, + # but in its first hours its 1d rate is understated (the documented young-series + # case) and it fires. c has no errors for 25h, then 9 a minute (0.15/s): over + # the floor with no baseline, so it fires, later than a because its 30m rate + # takes longer to pass the floor. d has no errors, then 3 a minute: a rise, but + # under the floor, so it never fires. - interval: 1m input_series: - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="a"}' @@ -106,10 +105,10 @@ tests: - eval_time: 1500m alertname: IrohDnsServerLookupErrors exp_alerts: [] - - eval_time: 1510m + - eval_time: 1518m alertname: IrohDnsServerLookupErrors exp_alerts: [] - - eval_time: 1530m + - eval_time: 1519m alertname: IrohDnsServerLookupErrors exp_alerts: - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} @@ -119,11 +118,51 @@ tests: - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} - exp_labels: {job: iroh-dns-server, instance: c, component: dns, severity: warning} - # Publishes, per minute. a takes updates and no-op puts until 60m, then none: no - # put in 30m from 90m, so it fires after the for (95m), keeps firing while the - # last put is in the 1d window, and resolves once it is not (25h). b never took - # a put, so it never fires. c takes no-op puts only, which count. d restarts at - # 40m (its counters reset) and goes on taking puts: no alert. + # The 3x and floor boundaries, the day's memory, and resolution. e answers 60 + # a minute for 25h, then 150 (2.5x its day): never fires. f goes to 240 (4x): + # fires. (3.5x would hold only for a while, as the rise enters the 1d window.) + # g has none, then 5 a minute (0.083/s, under the floor): never. h: none, then + # 7 a minute (0.117/s): fires. i goes from 3 to 60 a minute for 7h and still + # fires 6h in (a 6h baseline would have caught up). j goes to 60 a minute for + # an hour, then back to 3: it resolves once its 30m rate is under the floor. + - interval: 1m + input_series: + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="e"}' + values: '0+60x1500 90000+150x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="f"}' + values: '0+60x1500 90000+240x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="g"}' + values: '0x1500 5+5x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="h"}' + values: '0x1500 7+7x150' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="i"}' + values: '0+3x1500 4560+60x420' + - series: 'dns_server_dns_lookup_error_total{job="iroh-dns-server", instance="j"}' + values: '0+3x1500 4560+60x60 8160+3x120' + alert_rule_test: + - eval_time: 1560m + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: f, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: h, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: i, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: j, component: dns, severity: warning} + - eval_time: 1700m + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: i, component: dns, severity: warning} + - eval_time: 1860m + alertname: IrohDnsServerLookupErrors + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: i, component: dns, severity: warning} + + # Publishes, per minute. a takes puts until 60m, then none: its 30m increase is + # 0 from 89m, so it is pending at 92m, fires at 94m (the for), keeps firing + # while the last put is in the 1d window, and resolves once it is not (~25h). + # b never took a put, so it never fires. c takes updates until 60m and only + # no-op puts after: the no-op term keeps it quiet. d restarts at 40m (its + # counters reset) and goes on taking puts: no alert at 50m, with the reset + # inside the 30m window, nor later. - interval: 1m input_series: - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="a"}' @@ -135,18 +174,21 @@ tests: - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="b"}' values: '0x1660' - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="c"}' - values: '0x1660' + values: '0+2x60 120x1600' - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="c"}' - values: '0+5x1660' + values: '0x60 0+5x1600' - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="d"}' values: '0+2x40 0+2x1620' - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="d"}' values: '0+10x40 0+10x1620' alert_rule_test: + - eval_time: 50m + alertname: IrohDnsServerNoPublishes + exp_alerts: [] - eval_time: 92m alertname: IrohDnsServerNoPublishes exp_alerts: [] - - eval_time: 96m + - eval_time: 94m alertname: IrohDnsServerNoPublishes exp_alerts: - exp_labels: {job: iroh-dns-server, instance: a, component: dns, severity: warning} @@ -157,3 +199,39 @@ tests: - eval_time: 25h30m alertname: IrohDnsServerNoPublishes exp_alerts: [] + + # Each summed counter counts on its own, and resolution on recovery. u takes + # updates only (the steady state: every republish is an update): no alert. n + # takes no-op puts only and v updates only, both stopping at 60m: both fire, + # so neither term can be dropped from either arm. r stops at 60m and resumes at + # 120m: it fires, then resolves as soon as puts come back. + - interval: 1m + input_series: + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="u"}' + values: '0+5x200' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="u"}' + values: '0x200' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="n"}' + values: '0x200' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="n"}' + values: '0+5x60 300x140' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="v"}' + values: '0+5x60 300x140' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="v"}' + values: '0x200' + - series: 'dns_server_pkarr_publish_update_total{job="iroh-dns-server", instance="r"}' + values: '0+2x60 120x59 122+2x80' + - series: 'dns_server_pkarr_publish_noop_total{job="iroh-dns-server", instance="r"}' + values: '0+10x60 600x140' + alert_rule_test: + - eval_time: 100m + alertname: IrohDnsServerNoPublishes + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: n, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: r, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: v, component: dns, severity: warning} + - eval_time: 121m + alertname: IrohDnsServerNoPublishes + exp_alerts: + - exp_labels: {job: iroh-dns-server, instance: n, component: dns, severity: warning} + - exp_labels: {job: iroh-dns-server, instance: v, component: dns, severity: warning} From 30dc8b8d7c450d084f4a863c6870d1a84ff03f85 Mon Sep 17 00:00:00 2001 From: Ant Somers Date: Sun, 1 Nov 2026 15:56:37 +0300 Subject: [PATCH 3/3] fix(monitoring): iroh-dns-server records expire by signing time, and the cache outlives them Store eviction runs on the packet's signed timestamp (each republish re-signs, so about its last accepted put) and does not clear the lookup cache, so a cached key can keep resolving past the 7 days (Copilot review on #201). Co-Authored-By: Claude Opus 5.5 --- .../iroh-dns-server/dashboard-iroh-dns-server.json | 2 +- monitoring/iroh-dns-server/prometheus-alerts.yml | 10 ++++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json index d3a85513..ffcbc4c6 100644 --- a/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json +++ b/monitoring/iroh-dns-server/dashboard-iroh-dns-server.json @@ -448,7 +448,7 @@ "uid": "${DS_PROMETHEUS}" }, "title": "Record store per minute", - "description": "Signed packets inserted (a key first seen), updated (replaced by a newer one: every republish, so it tracks the update rate of the panel beside it) and expired (evicted 7 days after its last put) in the store. Removed is always 0 in 1.3.0.", + "description": "Signed packets inserted (a key first seen), updated (replaced by a newer one: every republish, so it tracks the update rate of the panel beside it) and expired (evicted 7 days after it was signed, about its last accepted put; eviction does not clear the lookup cache) in the store. Removed is always 0 in 1.3.0.", "type": "timeseries", "targets": [ { diff --git a/monitoring/iroh-dns-server/prometheus-alerts.yml b/monitoring/iroh-dns-server/prometheus-alerts.yml index 6012a5a7..58439fcb 100644 --- a/monitoring/iroh-dns-server/prometheus-alerts.yml +++ b/monitoring/iroh-dns-server/prometheus-alerts.yml @@ -129,7 +129,9 @@ groups: certificate (curl -v https:///healthz from outside), the nodes' decdn_discovery_pkarr_url and decdn_discovery_dns_origin, and whether - iroh_dns_server_pkarr_put_rate_limit refuses them. Stored - records keep resolving until they expire, 7 days after their - last put, but a node whose addresses change meanwhile is - resolved to the old ones. + iroh_dns_server_pkarr_put_rate_limit refuses them. The store + drops a record 7 days after the time it was signed (each + republish re-signs it, so about its last accepted put), and a + node whose addresses change meanwhile is resolved to the old + ones. A key already in the lookup cache can keep resolving + past that: eviction does not clear the cache.