[thiras]

What

decdn/decdn#2821 serves /metrics from process start and holds decdn_node_boot_ready at 0 until boot finishes. After that release, a 200 on /metrics no longer means the node has booted. Every path now uses one rule:

Ready = /metrics answers 200 and says exactly decdn_node_boot_ready 1, or has no decdn_node_boot_ready series at all. A sample this can't read (0.0, labelled, timestamped) counts as still booting.

v0.3.0 and v0.3.1 export no gauge and bind /metrics only after boot, so a 200 still means ready on them. That makes this safe to merge before the release that carries #2821. Hosts held on an exact v0.3.0 or v0.3.1 keep working.

• Role (decdn_node § Readiness): the probe keeps retrying a 200 that says 0. A timeout is still advisory. The warning now says which state the node ended in: still booting (it names DecdnNodeBootStuck and decdn_chain_boot_read_retries_total), or /metrics never answered. A non-200 answer still fails the run.
• decdn-compose health: the decdn-node metrics check becomes decdn-node boot. It is read from the gauge by a new boot_state() helper, which has unit tests.
• Chart: the default startupProbe is now exec decdn node health, the same check as the Compose healthcheck. The admin RPC starts only after boot, so the startup budget restarts a stuck boot again. That budget is now 10 minutes, matching upstream's own budget for retrying the boot chain reads. and the pod goes Ready only after boot. Readiness and liveness stay httpGet /metrics, because they run only after startup passes. The render now fails in two cases:
  • admin_port: 0 while the default startupProbe is in use.
  • Any probe with two handlers. Helm merges an override's httpGet into the default exec. The README explains how to null the default handler.
• Molecule: the stub binds at once with the gauge at 0, then sets it to 1 after 3 s. A marker file can switch it into one of three other modes: stuck, legacy (no gauge) or no-metrics.
  • default asserts that the probe ended on 1.
  • slow-readiness covers a stuck boot: the probe must retry the 200 for the whole window, and its warning must name the booting state. A new side_effect covers two more cases. A daemon that never binds must stay advisory: it warns, the unit stays up and is never restarted. A /metrics that answers 500 must fail the probe on its first attempt.
  • host-env covers a release without the gauge: one attempt, accepted.

Closes #204

Notes

• Chart upgrade caveat: a values file that copies the old httpGet startupProbe now fails the render (two handlers) until it sets startupProbe.exec: null. This is in the chart README, so the release notes should mention it.
• The collection and chart releases, and the Grafana republish, still wait for the decdn release that carries #2821 (the feat(monitoring): alert and panel for decdn_node_boot_ready #203 rollout).
• The CLI never runs its self-update check when stdout or stderr is not a terminal, so the exec probe makes no network calls.

After review (second commit)

Applied the four-agent review:

• Role: stricter gauge match; timeout: 5 on the probe; the warning prints the sample it saw, or the last error.
• Compose: boot_state reads labelled and timestamped samples, and no longer claims a gauge on a release that has none.
• Chart:
  • The admin_port guard now catches a shell-wrapped node health command and refuses a string command.
  • NOTES.txt and the README now describe the right ordering: the first ContentBlacklist sync is part of boot.
  • Startup budget raised to 10 minutes.
• Tests: a 500 path, warning-branch asserts, a restart-free check, stub timing made deterministic, and a fix for a possibly flaky check in host-env.

Verification

• make test-scripts, make lint-helm (new render checks and expect_fails), make lint, make lint-ansible: all pass.
• make molecule SCENARIOS='default slow-readiness host-env grafana-cloud': 4/4 pass. Both warning branches and the 500 rescue ran in slow-readiness.
• Ran the stub by hand in each mode and checked the body with curl.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

The slow-readiness check can race the restarted daemon’s socket binding and fail spuriously, so it needs bounded retries until /metrics returns 200 before checking the gauge.

1 open issue

• Wait for /metrics readiness after restarting the stub · New

What changed in this PR

Updates boot-readiness checks across Ansible, Compose, and Helm for nodes that expose /metrics before startup finishes, while preserving compatibility with releases without the boot-ready gauge.

Changes:

• Check decdn_node_boot_ready in Ansible and Compose, with clearer timeout diagnostics.
• Use the admin RPC for Helm’s startup probe and reject conflicting probe handlers.
• Extend unit tests and Molecule scenarios for ready, stuck, legacy, and unavailable-metrics states.

File	Description
compose/​tests/​test_decdn_compose.py	Tests boot-state parsing and legacy compatibility.
compose/​README.md	Documents gauge-aware health checks.
compose/​decdn-compose	Reports node boot readiness from metrics.
charts/​decdn-node/​values.yaml	Switches the default startup probe to admin RPC.
charts/​decdn-node/​tests/​render-test.sh	Checks startup commands and invalid probe configurations.
charts/​decdn-node/​templates/​_helpers.tpl	Validates probe handlers and admin RPC availability.
charts/​decdn-node/​README.md	Explains probe behavior and override migration.
ansible/​roles/​decdn_node/​tasks/​main.yml	Polls boot readiness and distinguishes timeout states.
ansible/​roles/​decdn_node/​README.md	Documents readiness and compatibility behavior.
ansible/​roles/​decdn_node/​defaults/​main.yml	Updates readiness comments; defaults remain unchanged.
ansible/​README.md	Updates the readiness scenario description.
ansible/​molecule/​slow-readiness/​verify.yml	Verifies a running node remains booting.
ansible/​molecule/​slow-readiness/​side_effect.yml	Exercises the unavailable-metrics timeout path.
ansible/​molecule/​slow-readiness/​prepare.yml	Configures the stub for a stuck boot.
ansible/​molecule/​slow-readiness/​molecule.yml	Shares inventory settings and adds side effects.
ansible/​molecule/​slow-readiness/​converge.yml	Asserts retries continue during a stuck boot.
ansible/​molecule/​host-env/​prepare.yml	Configures legacy gauge-less metrics.
ansible/​molecule/​host-env/​converge.yml	Verifies acceptance of gauge-less responses.
ansible/​molecule/​default/​files/​decdn-node-stub	Models delayed readiness and alternate boot states.
ansible/​molecule/​default/​converge.yml	Asserts the probe finishes with boot-ready set.
ansible/​Makefile	Updates the readiness scenario comment.

_{Review effort: Balanced}