[thiras]

Closes #202.

decdn/decdn#2821 binds the node's /metrics listener before the RPC preflight and boot chain reads and adds decdn_node_boot_ready (0 until the node runtime ready banner, then 1). A node stuck in boot then scrapes fine (up == 1), so DecdnNodeDown stops paging for it.

• DecdnNodeBootStuck (decdn-liveness, before DecdnRpcUnhealthy): max_over_time(decdn_node_boot_ready[15m]) == 0 for 10m, component: node, severity: warning, runbook #rpc-unreachable (that section covers boot). This is the recommended alert from adr/appendix-observability.md § Node / Process Metrics. It fires about 10m into a first boot and up to 25m after a ready node restarts.
• promtool tests: a first boot fires at exactly 10m. A lone 1 holds the alert off while it is in the window. A restart of a ready node fires 25m after the last 1. A crash loop (each failed scrape a staleness marker) keeps firing through its gaps, and a node that comes up resolves at its first 1. Mutation-checked: the bare gauge, min_over_time and keep_firing_for each fail the block.
• Two older rule comments (slash backfill, cache recency seeding) no longer claim their step runs before the first scrape.
• dashboard-chain.json: the "Watcher persistence and chain RPC" panel gets target E, min(decdn_node_boot_ready{…}), "min boot ready". Its description no longer says the listener binds after the boot reads.
• monitoring/README.md and the grafana-dashboards skill list the new tested rule.

DecdnRpcUnhealthy, DecdnNoActiveStreams and DecdnDhtRoutingTableEmpty are unchanged (see #202).

The readiness checks in the role, decdn-compose health and the chart probes also assume /metrics answers only after boot. Fixing them is #204, which ships in the same rollout.

Draft until decdn ships #2821. It merged after v0.3.1, so no release exports the gauge yet. This PR merges in the same rollout as that release, right after its bump PR. Then: release the chart, and republish the decdn-chain dashboard and the rule to Grafana.

Verified: make lint-helm (promtool check + unit tests) and make lint. To do before ready: run the name check against a node on the release.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

The unreleased decdn_node_boot_ready gauge still needs live metric-name validation, and rollout requires coordination with the decdn bump, readiness companion #204, chart release, and Grafana publishing.

No open issues 🎉

What changed in this PR

Adds boot-readiness monitoring for deCDN nodes that answer metrics requests before startup finishes, covering the monitoring gap described in #202.

Changes:

• Adds DecdnNodeBootStuck with tests for first boot, restart timing, crash-loop gaps, and recovery.
• Adds “min boot ready” to the chain dashboard and corrects comments about metrics availability during startup.
• Updates monitoring documentation and skill guidance. Rollout is coordinated with the upstream metric release; readiness-probe changes remain in #204.

File	Description
monitoring/​README.md	Lists the new alert’s unit-test coverage.
monitoring/​decdn-node/​prometheus-alerts.yml	Adds the boot-stuck alert and corrects startup-related comments.
monitoring/​decdn-node/​prometheus-alerts_test.yml	Tests alert timing, crash-loop persistence, and recovery.
monitoring/​decdn-node/​dashboard-chain.json	Displays minimum boot readiness across selected nodes.
.claude/​skills/​grafana-dashboards/​SKILL.md	Updates rule counts and the tested-alert list.

_{Review effort: Balanced}