Skip to content

readiness: /metrics answers during boot after decdn#2821; check decdn_node_boot_ready #204

Closed

Description

@thiras

What are you trying to do?

Keep the readiness checks honest once the decdn release that carries decdn/decdn#2821 ships.

#2821 binds the node's /metrics listener before the RPC preflight and the boot chain reads, and adds decdn_node_boot_ready (0 until the node runtime ready banner, then 1). Today every path treats "/metrics answers 200" as "boot finished". After #2821 that no longer holds:

  • Chart. charts/decdn-node/values.yaml (the probe comment) and README.md § Probes and lifecycle both say /metrics "binds only after startup completes". The startupProbe, readinessProbe and livenessProbe (all httpGet /metrics) then pass during boot:
    • the pod goes Ready while it is still booting;
    • the 5-minute startup budget no longer restarts a stuck boot, so only the node's own boot deadline ends one.
  • Role. ansible/roles/decdn_node/tasks/main.yml § Readiness and the role README treat the probe as advisory because the listener binds late. Its warning text says the same. A 200 during boot would pass the deploy without the node ever reaching ready.
  • Compose. decdn-compose health reports decdn-node metrics ok on any 200.
  • Molecule. ansible/molecule/default/files/decdn-node-stub and the slow-readiness scenario model the late bind, which becomes old-release behaviour.

What would help?

  1. Role readiness: wait for decdn_node_boot_ready 1 in the /metrics body, not just a 200. Rewrite the comment and the warning to match. Decide what a release without the gauge does; once the line's floor carries #2821, it can be treated as not ready.
  2. decdn-compose health: report boot-ready from the gauge, not from the status code. Add unit tests.
  3. Chart: kubelet httpGet cannot read the gauge's value. So say what the probes now prove (the listener is up), point to DecdnNodeBootStuck (feat(monitoring): alert and panel for decdn_node_boot_ready #203) for a stuck boot, and reconsider the startupProbe budget, possibly with an exec probe if the image has a client.
  4. Stub and molecule: have the stub serve /metrics early with decdn_node_boot_ready 0, then flip it to 1. Keep slow-readiness covering the slow boot.

Timing: same rollout as #203 and the decdn bump that carries #2821.

Deploy path

Ansible, Compose, Helm chart

Activity

  1. self-assigned this
    on Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions