test(molecule): split the slowest side effects into siblings

CONTRIBUTING.md

@@ -99,7 +99,8 @@ they are maintained here. See [its README](monitoring/README.md).
   `cloud-init-publisher` scenarios boot the real user-data templates, so they need
   network access to apt, PyPI and Galaxy. Keep each scenario well under its leg's
   timeout; when one grows, split it along its side effects (as `sponsord-install`,
-  `sponsord-onramp-nginx`, `sponsord-onramp-source`, `iroh-relay-gate`,
+  `sponsord-wallet`, `sponsord-onramp-nginx`, `sponsord-onramp-turnstile`, `sponsord-onramp-source`,
+  `iroh-relay-gate`, `iroh-relay-access`,
   `source-build-recovery` and the `validation-*` scenarios are), or move a play into
   a short sibling that already shares its converge. `source-build`, its two siblings
   and `sponsord-onramp-source` download rustup and a Rust toolchain from

ansible/Makefile

@@ -202,18 +202,20 @@ decommission:
 # /metrics probe timeout), grafana-cloud and grafana-cloud-token (opt-in Grafana
 # Cloud observability, host-provisioned vs inventory token), os-matrix (default's
 # plays on Debian 13, Ubuntu 24.04 and 26.04), lifecycle (network profile,
-# backup + decommission), sponsord and sponsord-install (the sponsord role on a host
-# with no node, through playbooks/sponsord.yml, on Debian 12 and Ubuntu 24.04; the
-# install-method, wallet and secret.env provenance side effects run in
-# sponsord-install), sponsord-onramp and sponsord-onramp-nginx (the onramp beside its
-# daemon, behind the real distro nginx; the proxy, Caddy retirement, ACME, gate-page and
-# inventory Turnstile side effects run in sponsord-onramp-nginx), sponsord-onramp-source
+# backup + decommission), sponsord, sponsord-install and sponsord-wallet (the sponsord
+# role on a host with no node, through playbooks/sponsord.yml, on Debian 12 and Ubuntu
+# 24.04; the install-method side effects run in sponsord-install, the wallet and
+# secret.env provenance ones in sponsord-wallet), sponsord-onramp, sponsord-onramp-nginx
+# and sponsord-onramp-turnstile (the onramp beside its daemon, behind the real distro
+# nginx; the proxy, Caddy retirement and ACME side effects run in sponsord-onramp-nginx,
+# the gate-page and inventory Turnstile ones in sponsord-onramp-turnstile),
+# sponsord-onramp-source
 # (that converge, then both sponsord roles built from source), sponsord-onramp-lifecycle
 # (that converge, then sponsord's backup and decommission), iroh-relay,
-# iroh-relay-certificate, iroh-relay-gate, iroh-relay-lifecycle and
+# iroh-relay-certificate, iroh-relay-gate, iroh-relay-access, iroh-relay-lifecycle and
 # iroh-relay-install (the relay through playbooks/iroh_relay.yml against a stub on
 # Debian 12 and Ubuntu 24.04: restarts, the fatal gate, the port guard; then the
-# certificate check, the gate's other refusals with the bind and access modes, its
+# certificate check, the gate's other refusals, the bind and access modes, its
 # decommission, and its release install from a local mirror), validation-iroh-relay
 # (the relay's bad knobs), iroh-dns-server, iroh-dns-server-certificate,
 # iroh-dns-server-gate, iroh-dns-server-install and iroh-dns-server-lifecycle (the
@@ -264,10 +266,10 @@ decommission:
 # rename cannot leave it silently stale.
 SCENARIOS ?= all
 JOBS ?=
-SLOW_FIRST ?= sponsord-onramp-source sponsord-install cloud-init source-build-recovery \
-	iroh-relay-gate cloud-init-publisher sponsord source-build sponsord-onramp \
-	sponsord-onramp-nginx decdn-gateway-lifecycle iroh-relay source-build-rollback \
-	iroh-relay-certificate \
+SLOW_FIRST ?= sponsord-onramp-source sponsord-install sponsord-wallet cloud-init \
+	source-build-recovery iroh-relay-gate iroh-relay-access cloud-init-publisher sponsord \
+	source-build sponsord-onramp sponsord-onramp-nginx sponsord-onramp-turnstile \
+	decdn-gateway-lifecycle iroh-relay source-build-rollback iroh-relay-certificate \
 	validation iroh-relay-install os-matrix grafana-cloud
 MOLECULE_LOCK_PREFIX ?= /tmp/decdn-devops-molecule
 molecule_env = SCENARIOS='$(SCENARIOS)' JOBS='$(JOBS)' SLOW_FIRST='$(SLOW_FIRST)' \

ansible/README.md

@@ -480,21 +480,24 @@ itself is Debian 12), `lifecycle` (a `decdn_network` profile with an override, b
 backup scopes decrypted and checked, a rejected and a real decommission),
 `sponsord` (`playbooks/sponsord.yml` on a host with no node: generated token, credential
 rotation restart, the fatal `/healthz` gate, the top-up hold guard, Alloy scraping sponsord and not the node;
-`grafana-cloud` covers sponsord co-located with a node), `sponsord-install` (the same
-converge, then release mode against a signed loopback mirror, treasury wallet
-generation and the `secret.env` provenance guards), `sponsord-onramp` (the onramp beside its daemon, behind the real distro
+`grafana-cloud` covers sponsord co-located with a node), `sponsord-install` and
+`sponsord-wallet` (the same converge, then release mode against a signed loopback
+mirror; and treasury wallet generation and the `secret.env` provenance guards),
+`sponsord-onramp` (the onramp beside its daemon, behind the real distro
 nginx with a self-signed certificate: the https chain, the client address nginx passes
 on, a Turnstile rotation restart, the fatal gates), `sponsord-onramp-nginx` (the same
 converge, then `none` mode, a broken upstream behind nginx, a refused foreign nginx
 site, the retirement of the role's earlier Caddy, ACME against an ACME server that
-does not answer, a custom gate page and the Turnstile secret from inventory),
+does not answer), `sponsord-onramp-turnstile` (the same converge, then a custom gate
+page and the Turnstile secret from inventory),
 `sponsord-onramp-source` (the same converge, then both sponsord roles built from a
 fixture repo with `install_method: source`), `sponsord-onramp-lifecycle` (the same converge, then
 sponsord's backup decrypted and checked, decommission refused for a held top-up and a
 bad confirmation, then a real one), `iroh-relay` (the relay against a stub: restarts,
-the fatal gate, the port guard), `iroh-relay-certificate` and `iroh-relay-gate` (the
-same converge, then the certificate check; and the other gate refusals, QAD off, the
-bind addresses and the access modes), `iroh-relay-install` and `iroh-relay-lifecycle`
+the fatal gate, the port guard), `iroh-relay-certificate`, `iroh-relay-gate` and
+`iroh-relay-access` (the same converge, then the certificate check; the other gate
+refusals and QAD off; and the bind addresses and the access modes),
+`iroh-relay-install` and `iroh-relay-lifecycle`
 (the same converge, then the release install and decommission), `iroh-dns-server`
 (the DNS server against a stub: restarts, the fatal gate, the port guard by address),
 `iroh-dns-server-certificate`, `iroh-dns-server-gate`, `iroh-dns-server-install` and

ansible/molecule/iroh-relay-access/molecule.yml

@@ -0,0 +1,94 @@
+---
+# The relay's knobs, split from ../iroh-relay-gate (itself split from ../iroh-relay)
+# so they run side by side: the same host shapes, inventory and converge (that
+# scenario's prepare, converge and verify, by path), then side_effect.yml here:
+# 0.0.0.0 and a specific IPv4 bind are rendered, bound and probed; the access modes,
+# and every way the inventory read of node IDs fails. Two platforms: two nodes
+# sharing an endpoint ID needs two hosts.
+#
+# The inventory below must stay identical to ../iroh-relay/molecule.yml's
+# (`make test-scripts` checks); the stub is the one it names.
+driver:
+  name: docker
+platforms:
+  - name: relay-access-molecule
+    # Same digest-pinned images as ../sponsord (repo convention).
+    image: geerlingguy/docker-debian12-ansible@sha256:a131e3b7a9a38dda40e98b49b39d6200ffab0ad853c714a513f5eea41bab9a25
+    pre_build_image: true
+    command: /usr/lib/systemd/systemd
+    privileged: true
+    cgroupns_mode: host
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
+    groups:
+      - iroh_relay_hosts
+  # systemd 255: StateDirectory= and the ambient capability on a newer systemd.
+  - name: relay-access-molecule-u2404
+    image: geerlingguy/docker-ubuntu2404-ansible@sha256:3394e335df58c961c00b78ae601ef30cdd7463b112f21b0c5f78674a70526302
+    pre_build_image: true
+    command: /usr/lib/systemd/systemd
+    privileged: true
+    cgroupns_mode: host
+    volumes:
+      - /sys/fs/cgroup:/sys/fs/cgroup:rw
+    groups:
+      - iroh_relay_hosts
+provisioner:
+  name: ansible
+  options:
+    skip-tags: baseline
+  env:
+    ANSIBLE_ROLES_PATH: "${MOLECULE_PROJECT_DIRECTORY}/roles"
+    ANSIBLE_COLLECTIONS_PATH: "${MOLECULE_PROJECT_DIRECTORY}/collections"
+    # Same pipelining note as ../default/molecule.yml.
+    ANSIBLE_PIPELINING: "true"
+  playbooks:
+    prepare: ../iroh-relay/prepare.yml
+    converge: ../iroh-relay/converge.yml
+    verify: ../iroh-relay/verify.yml
+  inventory:
+    group_vars:
+      iroh_relay_hosts:
+        iroh_relay_install_method: manual
+        iroh_relay_manual_bin_src: "${MOLECULE_PROJECT_DIRECTORY}/molecule/iroh-relay/files/iroh-relay-stub"
+        iroh_relay_hostname: relay.molecule.test
+        iroh_relay_acme_contact: ops@molecule.test
+        # A real ed25519 public key in iroh's hex form.
+        # The default allowlist mode: one ID listed here, plus the ID the stand-in
+        # `decdn` (prepare.yml) prints on each platform, every platform read as if
+        # it were one of the inventory's deCDN nodes (by every relay, across hosts).
+        iroh_relay_allowlist:
+          - 81d82536cb4c96ddbeab1f6bf4495d126c1942fb90b9ffb433d91d23314801e7
+        iroh_relay_allowlist_node_hosts: "{{ groups['iroh_relay_hosts'] }}"
+        # Every optional key rendered, so the stub checks each one's shape.
+        iroh_relay_accept_conn_limit: 50
+        iroh_relay_accept_conn_burst: 100
+        iroh_relay_client_rx_bytes_per_second: 4194304
+        iroh_relay_client_rx_max_burst_bytes: 8388608
+        iroh_relay_log_level: info,iroh_relay=debug
+        iroh_relay_certificate_check: fail
+        iroh_relay_certificate_retries: 5
+        iroh_relay_certificate_delay: 1
+        iroh_relay_readiness_retries: 10
+        iroh_relay_readiness_delay: 1
+        # 1s, not the default 5: the side effects run the role dozens of times. The
+        # crash-after-bind case (../iroh-relay-gate) sets its own 5s.
+        iroh_relay_readiness_settle: 1
+        decdn_grafana_cloud_enabled: true
+        grafana_alloy_install_method: manual
+        grafana_alloy_manual_bin_src: "${MOLECULE_PROJECT_DIRECTORY}/molecule/grafana-cloud/files/alloy-stub"
+        grafana_alloy_region: US
+        grafana_alloy_loki_url: https://logs-prod-xx.molecule.invalid/loki/api/v1/push
+verifier:
+  name: ansible
+scenario:
+  test_sequence:
+    # Leading destroy: see ../default/molecule.yml. No idempotence step: the
+    # converge is ../iroh-relay's, whose idempotence runs there.
+    - destroy
+    - create
+    - prepare
+    - converge
+    - side_effect
+    - verify
+    - destroy

ansible/molecule/iroh-relay-access/side_effect.yml

@@ -0,0 +1,102 @@
+---
+# What a green converge + idempotence cannot prove, on ../iroh-relay's converge (split
+# from ../iroh-relay-gate so the two run side by side):
+#   5. 0.0.0.0 and a specific IPv4 address: rendered, bound, and probed there (the
+#      certificate check is "fail" in this inventory).
+#   6. The allowlist: every relay reads every platform's node ID; a node with no
+#      identity, a failing `whoami`, an unreachable node, a host outside the
+#      inventory and two nodes sharing an ID each fail tasks_from node-ids; the
+#      read works in check mode; with the inventory read off only the listed IDs
+#      render; the "everyone" and "denylist" modes render as such. (A role run that
+#      skipped the read is refused: ../validation-iroh-relay.) Two nodes sharing an
+#      ID needs two hosts, hence this scenario's two platforms.
+# The include files are ../iroh-relay's. The host is left as converged, with the
+# molecule CA trusted, for verify (../iroh-relay's).
+- name: Side effect — bind addresses, access modes
+  hosts: all
+  become: true
+  tasks:
+    # As playbooks/iroh_relay.yml does before the role: facts the converge run set
+    # do not carry over to this run.
+    - name: Read the inventory nodes' endpoint IDs, as the playbook does
+      ansible.builtin.include_role:
+        name: iroh_relay
+        tasks_from: node-ids
+
+    # --- 5. non-wildcard bind addresses ------------------------------------------------
+    # The certificate check is "fail" in this inventory, so a probe aimed at the
+    # wrong address fails these converges.
+    - name: Converge on each IPv4 bind address
+      ansible.builtin.include_tasks: ../iroh-relay/includes/bind-address.yml
+      loop:
+        - 0.0.0.0
+        - "{{ ansible_facts['default_ipv4']['address'] }}"
+      loop_control:
+        loop_var: _ir_bind
+
+    # --- 6. the access modes -----------------------------------------------------------
+    - name: Read the node IDs while one stand-in node misbehaves
+      ansible.builtin.include_tasks: ../iroh-relay/includes/node-ids-fail.yml
+      loop:
+        - {marker: molecule-no-identity, failed_at: Require an endpoint ID from every deCDN node}
+        - {marker: molecule-whoami-fails, failed_at: Require an endpoint ID from every deCDN node}
+        - {marker: molecule-shared-id, failed_at: Refuse deCDN nodes that share an endpoint ID}
+      loop_control:
+        loop_var: _ir_case
+
+    # A node that cannot be reached is reported by name, not as the relay host.
+    - name: Add a deCDN node nobody can reach
+      ansible.builtin.add_host:
+        name: molecule-unreachable-node
+        ansible_connection: ssh
+        ansible_host: 127.0.0.1
+        ansible_port: 1
+        ansible_ssh_common_args: -o ConnectTimeout=3
+
+    - name: Read the node IDs of an unreachable node and of a host outside the inventory
+      ansible.builtin.include_tasks: ../iroh-relay/includes/node-ids-fail.yml
+      loop:
+        - hosts: [molecule-unreachable-node]
+          failed_at: Require an endpoint ID from every deCDN node
+        - hosts: [molecule-not-an-inventory-host]
+          failed_at: Validate iroh_relay_allowlist_node_hosts
+      loop_control:
+        loop_var: _ir_case
+
+    - name: Read the inventory nodes' IDs in check mode
+      ansible.builtin.include_role:
+        name: iroh_relay
+        tasks_from: node-ids
+        apply:
+          check_mode: true
+
+    - name: Require the check-mode read to have read every node
+      ansible.builtin.assert:
+        that:
+          - _iroh_relay_inventory_node_ids_read
+          - _iroh_relay_inventory_node_ids | sort == groups['iroh_relay_hosts'] | map('hash', 'sha256') | sort
+        fail_msg: "Check mode read {{ _iroh_relay_inventory_node_ids }}."
+
+    - name: Converge in each other access mode
+      ansible.builtin.include_tasks: ../iroh-relay/includes/access-mode.yml
+      loop:
+        - {access: everyone, allowlist: [], denylist: [], want: '"everyone"'}
+        # The inventory read off: only the listed ID, none of the nodes'.
+        - access: allowlist
+          allowlist: [81d82536cb4c96ddbeab1f6bf4495d126c1942fb90b9ffb433d91d23314801e7]
+          denylist: []
+          from_inventory: false
+          want: '{ allowlist = ["81d82536cb4c96ddbeab1f6bf4495d126c1942fb90b9ffb433d91d23314801e7"] }'
+        - access: denylist
+          allowlist: []
+          denylist: [08ebd644cd0bd76975a53e7ff96b1eacd14dd4f61672f3f54c7c8bb19fbf2a08]
+          want: '{ denylist = ["08ebd644cd0bd76975a53e7ff96b1eacd14dd4f61672f3f54c7c8bb19fbf2a08"] }'
+      loop_control:
+        loop_var: _ir_mode
+
+    - name: Converge back on the inventory's settings
+      ansible.builtin.include_role:
+        name: iroh_relay
+
+    - name: Check the relay is back on a current record
+      ansible.builtin.include_tasks: ../iroh-relay/includes/record-is-current.yml

ansible/molecule/iroh-relay-gate/molecule.yml

@@ -1,13 +1,12 @@
 ---
-# The relay's other gate refusals and its knobs, split from ../iroh-relay so the two
-# run side by side: the same host shapes, inventory and converge (that scenario's
-# prepare, converge and verify, by path), then side_effect.yml here: a relay that
-# skips its QUIC socket or crashes after binding FAILS the gate; the port guard
-# passes the unit's own listeners while it restarts or has no MainPID; a missing
-# record restarts a running relay; QAD off renders and binds no udp/7842, and a
-# squatter there is refused only with QAD on; 0.0.0.0 and a specific IPv4 bind are
-# rendered, bound and probed; the access modes, and every way the inventory read of
-# node IDs fails. Two platforms: two nodes sharing an endpoint ID needs two hosts.
+# The relay's other gate refusals, split from ../iroh-relay so the two run side by
+# side: the same host shapes, inventory and converge (that scenario's prepare,
+# converge and verify, by path), then side_effect.yml here: a relay that skips its
+# QUIC socket or crashes after binding FAILS the gate; the port guard passes the
+# unit's own listeners while it restarts or has no MainPID; a missing record
+# restarts a running relay; QAD off renders and binds no udp/7842, and a squatter
+# there is refused only with QAD on. Bind addresses and the access modes are
+# ../iroh-relay-access, on the same two platforms.
 #
 # The inventory below must stay identical to ../iroh-relay/molecule.yml's
 # (`make test-scripts` checks); the stub is the one it names.