From 8680a2d7e3fa5fa6bd59a3ba8942a3d71a5856b9 Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 19 Oct 2026 14:08:21 +0200 Subject: [PATCH 1/6] feat(health): the deploy host asks /healthz on a schedule, and it pages MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `probe-loopback` was dispatch-only, so the three states only it can see — a stopped snapshot worker, a stale ingest behind a live tunnel, a dead dashboard process — had no observer at all since the Pi timer was retired. It now runs on the same `*/15` schedule as `probe-ingest` and pages, reproducing that job's accepted shape rather than inventing one: - its own production marker, `cotel-healthz-host`, free of the ingest marker, the drill markers and the script's default (which belonged to the retired timer and may still have an alert standing under it); - `PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`, so one outage mints exactly one ticket instead of up to 96 a day; - the recovery wake on green, which is what re-arms the pager once the dedup window is off; - an hourly `PC_CALL_ID` bucket, so a long-standing alert does not mint a recovery wake every green tick; - a red confirmed inside the run by a second probe 60 seconds later: a push to `main` is a deploy here, and a tick landing in the container restart would otherwise page for a routine deploy. The ingest staleness threshold moves to 24 hours on the schedule and stays at 6 on a dispatch. An hourly span histogram over a week of production shows ordinary overnight silence reaching six and seven hours, so a scheduled pager on the old window would have fired several times that week on nothing; the longest silence ever measured here, 12 hours, was the real tunnel outage. The snapshot window is untouched — that worker ticks independently of traffic. Docs follow the behaviour: the workflow header, the vantage-point table, the drill section and the pager environment tables no longer say `/healthz` is unwatched; `duckdb-snapshots.md` no longer says nobody watches the snapshot worker; ADR-0026 records the decision, the measurement and the second marker, and ADR-0025 gets one line noting which of its consequences this lifts. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- .github/workflows/health-probe.yml | 119 ++++-- docs/.vitepress/config.js | 1 + ...5-best-effort-ingest-watcher-in-actions.md | 6 + ...6-scheduled-loopback-probe-with-a-pager.md | 164 ++++++++ docs/decisions/index.md | 1 + docs/operations/duckdb-snapshots.md | 35 +- docs/operations/health-probe.md | 354 +++++++++++------- 7 files changed, 499 insertions(+), 181 deletions(-) create mode 100644 docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index ebbf54f..a8e7124 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -1,27 +1,33 @@ name: Health probe -# One job here is scheduled, and it is a **best-effort watcher, not -# monitoring**: `probe-ingest` asks the public OTLP ingest URL from outside -# every network cotel lives in. GitHub schedules public repositories on a -# best-effort basis and *drops* ticks rather than delaying them — measured in -# this repository, one run out of the nine expected over nine hours — so the -# detection delay it buys is unbounded. It is strictly better than nothing, -# and that is the whole of what it promises. See -# docs/operations/health-probe.md and ADR-0025. +# Two jobs here are scheduled, and both are **best-effort watchers, not +# monitoring**. GitHub schedules public repositories on a best-effort basis and +# *drops* ticks rather than delaying them — measured in this repository, one run +# out of the nine expected over nine hours — so the detection delay either one +# buys is unbounded. They are strictly better than nothing, and that is the +# whole of what they promise. See docs/operations/health-probe.md, ADR-0025 and +# ADR-0026. # -# That job is `ubuntu-latest` and must never be self-hosted. An observer inside -# the network it observes shares the uplink and the power outlet with it and -# goes down alongside it, which is why the 10-minute timer that used to own the -# production marker was retired. +# `probe-ingest` asks the public OTLP ingest URL from outside every network +# cotel lives in. It is `ubuntu-latest` and must never be self-hosted: an +# observer inside the network it observes shares the uplink and the power outlet +# with it and goes down alongside it, which is why the 10-minute timer that used +# to own the production `/healthz` marker was retired. # -# The other two jobs are dispatch-only drills of the pager and page *drill* -# markers, never production. +# `probe-loopback` asks `/healthz` on the deploy host, which is the only vantage +# point that can see a 503 database, a stale ingest and a missing snapshot. It +# must stay self-hosted for the same reason, and it pays for that with the +# opposite blindness: it cannot report its own host being down. The two +# blindnesses are complementary, which is why neither job is sufficient alone. +# +# `probe-edge` remains a dispatch-only drill and pages a *drill* marker, never +# production. on: schedule: - # The cadence is free — a dropped tick costs nothing, a kept one costs one - # HTTP request — so it is set to the floor a reader would tolerate rather - # than to a budget. It does not bound the detection delay; nothing that - # runs here can. + # The cadence is free — a dropped tick costs nothing, a kept one costs a + # couple of HTTP requests and a minute on a runner we own — so it is set to + # the floor a reader would tolerate rather than to a budget. It does not + # bound the detection delay; nothing that runs here can. - cron: "*/15 * * * *" workflow_dispatch: inputs: @@ -184,38 +190,63 @@ jobs: probe-loopback: name: Probe /healthz on the deploy host - if: github.event_name == 'workflow_dispatch' + if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' runs-on: [self-hosted, flopsstuff, docker] # This job cannot report its own host being down: with the runner offline # the run sits in the queue, and timeout-minutes does not bound queue time # — it starts counting once a runner picks the job up. So an absent host - # produces no colour at all, which is why the Pi timer probes this host - # over the LAN instead. The cap only bounds a job that is running and stuck - # (6h by default), and the concurrency group keeps a re-dispatch superseding - # the last queued attempt instead of stacking them. + # produces no colour at all, which is what `probe-ingest` is for. The cap + # only bounds a job that is running and stuck (6h by default), and the + # concurrency group keeps each tick superseding the last queued attempt + # instead of stacking them. timeout-minutes: 5 concurrency: group: health-probe-loopback cancel-in-progress: true env: HEALTHZ_URL: ${{ inputs.loopback_url || 'http://127.0.0.1:8080/healthz' }} - STALE_AFTER_SECONDS: "21600" + # A day on the schedule, six hours by hand. Measured over a week on this + # instance, ordinary ingest silence reaches six and seven hours overnight, + # so a 6h window would page a quiet night several times a week; the + # longest silence ever measured here, 12h, was a real outage. A dispatch + # keeps the tighter window because a human is reading that answer. The + # snapshot window is unaffected — that worker ticks on its own timer and + # does not care how much traffic arrives. + STALE_AFTER_SECONDS: ${{ github.event_name == 'schedule' && '86400' || '21600' }} CONNECT_TIMEOUT: "15" PC_API_URL: ${{ secrets.PAPERCLIP_API_URL }} PC_API_TOKEN: ${{ secrets.PAPERCLIP_API_TOKEN }} PC_COMPANY_ID: ${{ secrets.PAPERCLIP_COMPANY_ID }} # A dispatched run is an exercise and gets its own dedup marker, so a # drill can neither mint an alert that reads like an outage nor act on a - # standing real one. Set explicitly rather than left to the script - # default: inheriting the default is what made the drills invisible. - PC_ORIGIN_ID: cotel-health-probe-drill + # standing real one. The scheduled marker is new rather than the script's + # default: the default belonged to the retired Pi timer and may still have + # an alert standing under it, which this job must not inherit. + PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-healthz-host' || 'cotel-health-probe-drill' }} + # These two are a pair. `0` switches the staleness window off, so an open + # alert is always the dedup target and one outage mints exactly one + # ticket; `PC_STILL_RED_WAKE=0` drops the per-tick wake that would + # otherwise land 96 times a day while that ticket stands. What it costs is + # that the recovery wake is now the only thing that re-arms this watcher — + # an alert nobody closes makes the next outage silent. + PC_ALERT_MAX_AGE_H: "0" + PC_STILL_RED_WAKE: "0" GITHUB_RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} steps: - uses: actions/checkout@v5 + # A coarse bucket rather than this run id, which is the default. With the + # staleness window off an alert can stand for a long time, and a per-run + # key would make every green tick in the meantime mint another recovery + # wake — four an hour. A dispatch keeps the run id: each drill is its own + # call and wants its own recovery. + - name: Bucket the recovery key for the scheduled run + if: github.event_name == 'schedule' + run: echo "PC_CALL_ID=healthz-host-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" + # The pager is the only thing here that wakes anyone, and this runner is # a developer Mac rather than a managed image. Check its interpreters on - # every run, so a missing one surfaces during a drill rather than when an + # every run, so a missing one surfaces on a green tick rather than when an # incident needs them. - name: Check pager prerequisites run: | @@ -231,12 +262,26 @@ jobs: set -o pipefail mkdir -p "$RUNNER_TEMP" PROBE_OUT="$RUNNER_TEMP/probe-loopback.out" - set +e - scripts/probe-healthz.sh | tee "$PROBE_OUT" - rc=${PIPESTATUS[0]} - set -e + probe() { + set +e + scripts/probe-healthz.sh | tee "$PROBE_OUT" + PROBE_RC=${PIPESTATUS[0]} + set -e + } + # A push to `main` is a deploy here and the container restarts for a + # few seconds, so a tick landing in that window sees a dead process + # and would otherwise mint a ticket for a routine deploy. The streak + # lives inside this one run because there is nowhere between runs to + # keep it: ticks are dropped, so "two runs in a row" is not a quantity + # this job can count. + probe + if [ "$PROBE_RC" != "0" ]; then + echo "first probe exited ${PROBE_RC}; looking again in 60s and paging only if that is red too" + sleep 60 + probe + fi echo "PROBE_OUT=$PROBE_OUT" >> "$GITHUB_ENV" - echo "PROBE_RC=$rc" >> "$GITHUB_ENV" + echo "PROBE_RC=$PROBE_RC" >> "$GITHUB_ENV" # The pager puts this in the alert description, as the command the # woken agent re-runs. This job scopes it to the step above. echo "HEALTHZ_URL=$HEALTHZ_URL" >> "$GITHUB_ENV" @@ -244,7 +289,7 @@ jobs: # Paging failures are a warning, never the job's verdict: this job's # red/green must mean "production is red/green" and nothing else. - name: Page Paperclip on red - if: env.PROBE_RC != '0' && inputs.page + if: env.PROBE_RC != '0' && (github.event_name == 'schedule' || inputs.page) env: CF_ACCESS_CLIENT_ID: ${{ secrets.CF_ACCESS_CLIENT_ID }} CF_ACCESS_CLIENT_SECRET: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} @@ -252,8 +297,12 @@ jobs: scripts/page-cotel-health.sh raise "$PROBE_OUT" \ || echo "::warning title=Pager failed::could not raise the Paperclip alert; nobody was woken for this red probe" + # Not optional, even though this watcher only has to detect failure: with + # the staleness window off, the open alert dedups every later red tick, so + # until it is closed the *next* outage wakes nobody. One wake on recovery + # is what re-arms the pager. - name: Wake the alert's assignee on green - if: env.PROBE_RC == '0' && inputs.page + if: env.PROBE_RC == '0' && (github.event_name == 'schedule' || inputs.page) env: CF_ACCESS_CLIENT_ID: ${{ secrets.CF_ACCESS_CLIENT_ID }} CF_ACCESS_CLIENT_SECRET: ${{ secrets.CF_ACCESS_CLIENT_SECRET }} diff --git a/docs/.vitepress/config.js b/docs/.vitepress/config.js index f6d49ad..0b568c6 100644 --- a/docs/.vitepress/config.js +++ b/docs/.vitepress/config.js @@ -66,6 +66,7 @@ export default defineConfig({ { text: 'ADR-0023 - Production Database Snapshots', link: '/decisions/0023-production-database-snapshots' }, { text: 'ADR-0024 - Off-Host Snapshot Copy is a Pi-Side Pull', link: '/decisions/0024-off-host-snapshot-copy-pi-pull' }, { text: 'ADR-0025 - Best-Effort Ingest Watcher in Actions', link: '/decisions/0025-best-effort-ingest-watcher-in-actions' }, + { text: 'ADR-0026 - The Deploy Host Asks /healthz on a Schedule', link: '/decisions/0026-scheduled-loopback-probe-with-a-pager' }, ], }, ], diff --git a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md index f210034..132ebbd 100644 --- a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md +++ b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md @@ -131,6 +131,12 @@ bound, and the first person to rely on that is the one holding a silent outage. restore point is found only by hand**, by running `scripts/probe-healthz.sh` from any machine on the LAN. That is the price, recorded rather than discovered. +- **That price was paid on 2026-10-19 and this bullet no longer holds.** The CEO + granted the pager, and + [ADR-0026](./0026-scheduled-loopback-probe-with-a-pager) puts `probe-loopback` + on the same schedule with one. Everywhere this record says "nothing asks + `/healthz` on a schedule now" — the bullet above and the stale-ingest bullet + before it — read ADR-0026 instead. The rest of this record stands as written. - **Watching the worker is only half of watching the restore point, and the other half is not a watcher question.** What a lost snapshot worker actually costs is bounded by what the off-host copy holds, and that copy decays on its diff --git a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md new file mode 100644 index 0000000..8f9f429 --- /dev/null +++ b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md @@ -0,0 +1,164 @@ +# ADR 0026 — the deploy host asks `/healthz` on a schedule, and it pages + +**Date:** 2026-10-19 +**Status:** Accepted +**Deciders:** Prospero (CEO, granting board-facing output), Daedalus (CTO, thresholds and implementation) + +--- + +## Context + +[ADR-0025](./0025-best-effort-ingest-watcher-in-actions) put a best-effort +watcher on the **public OTLP ingest** and recorded, as a price rather than an +oversight, that it is blind to everything behind the auth boundary. Three states +share that blindness, and they are the three nothing has observed since the Pi +timer was retired on 2026-10-14: + +- **A snapshot worker that has stopped producing restore points.** This is the + one with an irreversible price: what is lost is lost, and the off-host mirror + decays on its own schedule ([ADR-0024](./0024-off-host-snapshot-copy-pi-pull)'s + pull has none). +- **A stale ingest behind a live tunnel.** The instance answers 401 from outside + while accepting no spans, or while its DuckDB is unreadable. +- **A dead dashboard.** `cotel.aignite.pl` is a different host, and the edge + probe for it cannot get past Cloudflare Access. + +No external vantage point can distinguish any of them from health, because from +outside all three look exactly like a healthy instance refusing an invalid +bearer. The only vantage point that sees them is the deploy host itself, and +this workflow already has it: `probe-loopback` asks +`http://127.0.0.1:8080/healthz` from the self-hosted runner, needs no +credential, and classifies the `snapshot` object from `/api/v1/health`. + +It was `workflow_dispatch`-only. ADR-0025 framed the remaining choice as binary +and not an implementation detail: scheduling that job *without* a pager is the +appearance of detection rather than detection, and scheduling it *with* one +re-adds board-facing output that was deliberately curated away when the previous +watcher was retired. That made it the CEO's call, and the CEO granted it. + +## Decision + +**`probe-loopback` runs on the same `*/15` schedule as `probe-ingest`, and it +pages — with its own dedup marker, the dedup pair that bounds one outage to one +ticket, an in-run red confirmation, and a measured staleness threshold.** + +Six things had to be true together; the first five copy `probe-ingest`, whose +shape the board accepted in the same form. + +1. **Own marker: `[cotel-healthz-host]`.** New rather than reused. It must not + collide with `[cotel-ingest-edge]` — on one marker the two halves would + fight, a green tick on one asking for the close of an alert the other had + just raised — nor with the drill markers, nor with the script's default + `[cotel-health-probe]`, which belonged to the retired Pi timer and may still + have an alert standing under it that this job must not inherit. +2. **`PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`.** The staleness window + off means an open alert is always the dedup target, so one outage mints + exactly one ticket; the second drops the per-tick wake. Without the pair, + one outage is up to 96 tickets a day. +3. **The recovery wake stays on.** It is not optional even though this watcher + only has to detect failure: with the window off, the open alert dedups every + later red tick, so until it is closed the *next* outage wakes nobody. + ADR-0025 records "we only need failure, so drop the green path" as the + tempting mistake, and it silently disarms the watcher. +4. **A coarse hourly recovery key, `healthz-host-`.** `PC_CALL_ID` + feeds the recovery wake's idempotency key and defaults to the run id. With + the staleness window off an alert can stand for a long time, and a per-run + key would make every green tick in the meantime mint another wake — 96 + alarms a day on one open alert. +5. **A red is confirmed inside the run**, by a second probe 60 seconds later. + The reason here is concrete rather than theoretical: a push to `main` is a + deploy, the container is down for a few seconds while it restarts, and a tick + landing in that window would otherwise mint a ticket for a routine deploy. + The streak cannot live between runs — dropped ticks mean "two runs in a row" + is not a quantity this job can count. +6. **The ingest staleness threshold is 24 hours on the schedule, 6 by hand.** + Measured, not assumed; see below. + +A dispatch keeps everything it had: the drill marker, the run id as the call id, +and the 6h threshold. Only the scheduled event gets the production marker and +the longer window, by the same `github.event_name == 'schedule'` conditional +`probe-ingest` already uses. + +### Why 24 hours, and where the number came from + +`scripts/probe-healthz.sh` exits **3** when `newest_span_age_seconds` exceeds +`STALE_AFTER_SECONDS`, and the job shipped with 6 hours — a threshold written +for a *dispatch*, where a human reads the answer. An hourly span histogram +pulled from production over one week (`/api/v1/history?granularity=hour&range=week`) +says 6 hours is not an assertion about this instance: + +``` +gap 12h 2026-10-14 09:00 -> 21:00 (the tunnel outage — a real red) +gap 6h 2026-10-17 04:00 -> 10:00 +gap 6h 2026-10-15 22:00 -> 04:00 +gap 6h 2026-10-15 10:00 -> 16:00 +``` + +Ordinary quiet nights on this box reach six and seven hours of ingest silence. +With a 6h window a scheduled pager would mint "stale ingest" tickets several +times a week on nothing at all — precisely the noise the board retired the +previous watcher for. A full day with no span is an assertion: the longest +silence ever measured here, 12 hours, was a genuine outage. So exit 3 stays a +paging class, with a threshold that has evidence under it. + +This does not touch `SNAPSHOT_STALE_AFTER_SECONDS` (12h). The snapshot worker +ticks on its own timer and does not care how much traffic arrives, so its window +is independent of the instance's traffic shape. + +## Consequences + +- **This job cannot report its own host being down, and nothing in it can be + made to.** With the runner offline the run sits in the queue, and + `timeout-minutes` does not bound queue time — it starts counting when a runner + picks the job up. An absent host therefore produces no colour at all, and the + only trace is a run in `queued`, which is also what a busy runner looks like. + That external failure is `probe-ingest`'s to catch, and it is why **neither + job is sufficient alone**: their blindnesses are complementary, one inside the + network and one outside it, which is also why neither may be moved to the + other's runner. +- **The three unwatched states above are now watched, on a best-effort + schedule.** The detection delay is unbounded for the same reason ADR-0025's is: + GitHub drops scheduled ticks on public repositories rather than delaying them. + Whether it ran at all is a measurement, not an assumption — + `gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule` + and empty output is the answer rather than an error. +- **A second production marker exists, and the two must stay separate.** + `[cotel-healthz-host]` and `[cotel-ingest-edge]` watch different things from + different places and can be red independently: the tunnel outage of 2026-10-14 + was red on the ingest marker while `/healthz` answered 200 on the LAN + throughout. Merging them would make each half's green tick close the other + half's alert. A reader seeing both open is looking at two findings, not a + duplicate. +- **Two production pagers can now mint tickets, so the board-facing cost + roughly doubles** at the ceiling of one ticket per outage per marker. That + ceiling is what the dedup pair buys, and it is the whole reason the pair is + not optional. +- **Recovery is load-bearing twice over.** An alert nobody closes disarms its + marker until someone does, and now there are two markers to keep clear. The + standing rule applies to both: the alert closes when its assignee next wakes + on a green tick, not the instant the probe recovers + ([ADR-0021](./0021-recovery-wakes-the-alerts-assignee)). +- **A deploy no longer pages, and that cost 60 seconds.** The in-run second look + means a red tick takes about a minute longer to page than it would otherwise. + That is the right trade against a ticket for every deploy that happens to + overlap a tick. +- **The pager's default re-probe hint is finally correct for one caller.** The + default text in `scripts/page-cotel-health.sh` tells the woken agent to + dispatch the loopback half, which is wrong for every *other* caller because + that runner may be the dead host. For an alert raised **by** this job the host + is demonstrably up — it just ran the probe — so the default stands and this + job overrides nothing. Any new caller still has to override it. +- **`probe-edge` stays dispatch-only**, and not on technical grounds: the Access + service token is still refused on `cotel.aignite.pl`, and asking again was + made conditional on an outage of the dashboard host specifically while the + ingest name is live. The dashboard is nonetheless no longer wholly unwatched — + a dead dashboard process shows up on this job's `/healthz`, from inside the + host; what remains unwatched is its path through Cloudflare. +- **ADR-0025's blind half is closed**, and its consequence bullets that read + "nothing asks `/healthz` on a schedule now" are lifted by this record. Its + other half stands: the off-host snapshot mirror still has no schedule, so a + green probe here is not a statement about recoverability. Read + `scripts/pull-snapshots.sh --status` before making one. +- **No new credential, and nothing for the board to do.** This job already had + every secret it needs; the change is an event name, a marker, three + environment variables and one more probe call. diff --git a/docs/decisions/index.md b/docs/decisions/index.md index 2e5ca46..54f194e 100644 --- a/docs/decisions/index.md +++ b/docs/decisions/index.md @@ -33,3 +33,4 @@ New ADRs go in this directory as `NNNN-short-title.md`, numbered sequentially. | [ADR-0023](./0023-production-database-snapshots) | Snapshots: `EXPORT DATABASE` to Parquet, on a timer, from inside cotel | Accepted | | [ADR-0024](./0024-off-host-snapshot-copy-pi-pull) | The off-host copy is a Pi-side pull, and it cannot be an rsync of the volume | Accepted | | [ADR-0025](./0025-best-effort-ingest-watcher-in-actions) | cotel is watched by a best-effort ingest probe in GitHub Actions | Accepted | +| [ADR-0026](./0026-scheduled-loopback-probe-with-a-pager) | The deploy host asks `/healthz` on a schedule, and it pages | Accepted | diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 9e0bdc5..9a5edc8 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -78,21 +78,26 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ ## Who is watching it -**Nobody is, as of 2026-10-14.** `scripts/probe-healthz.sh` reads `/healthz` -first and, when that is green, asks `/api/v1/health` and classifies the -`snapshot` object — a red verdict pages under the `[cotel-health-probe]` -marker, as an issue whose assignment wakes an agent. That still works, from any -machine on the LAN, but **no schedule runs it**: the 10-minute timer that did -was retired by board order, and the one scheduled watcher cotel has left asks -the public ingest URL only. See -[Production health probes](./health-probe#what-this-watcher-cannot-see). Until -something schedules this probe again, a stopped snapshot worker is found by -running the command above by hand. - -That is a standing price, not a pending task: -[ADR-0025](../decisions/0025-best-effort-ingest-watcher-in-actions.md) records why -the only job that could watch this on a schedule is still dispatch-only, and -what has to be granted before it is not. +**The `probe-loopback` job in `.github/workflows/health-probe.yml`, every 15 +minutes, best-effort, since 2026-10-19.** `scripts/probe-healthz.sh` reads +`/healthz` first and, when that is green, asks `/api/v1/health` and classifies +the `snapshot` object — a red verdict pages under the `[cotel-healthz-host]` +marker on a schedule, as an issue whose assignment wakes an agent. It runs on +the self-hosted runner on the deploy host, which is the only vantage point from +which `/api/v1/health` is reachable at all. The same probe still works by hand +from any machine on the LAN. See +[Production health probes](./health-probe#the-vantage-points) and +[ADR-0026](../decisions/0026-scheduled-loopback-probe-with-a-pager.md). + +Two caveats on what that buys. GitHub **drops** scheduled ticks on public +repositories rather than delaying them, so the detection delay is unbounded and +a green workflow file is not evidence anything ran — measure it with +`gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule`. +And a green verdict here is a statement about the *worker*, not about +recoverability: the off-host mirror has no schedule of its own and decays +independently, so read `scripts/pull-snapshots.sh --status` before concluding +anything from it +([ADR-0024](../decisions/0024-off-host-snapshot-copy-pi-pull.md)). The claim is deliberately **not** on `/healthz`. That endpoint is the container liveness contract: `cotel --healthcheck`, the Docker `HEALTHCHECK` and diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 37f60ba..bff9806 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -1,16 +1,28 @@ # Production health probes -One scheduled watcher asks production one question: the `probe-ingest` job in -`.github/workflows/health-probe.yml` requests the **public OTLP ingest URL** -from a GitHub-hosted runner every 15 minutes, and pages a Paperclip agent when -two looks a minute apart are both red. - -**It is a best-effort watcher, not monitoring.** GitHub schedules public +Two scheduled watchers ask production two different questions, both in +`.github/workflows/health-probe.yml`, both every 15 minutes, and both paging a +Paperclip agent when two looks a minute apart are red: + +- **`probe-ingest`** requests the **public OTLP ingest URL** from a + GitHub-hosted runner — production as the internet sees it. +- **`probe-loopback`** requests **`/healthz` on the deploy host** from the + self-hosted runner that lives there — the process, the database, ingest + freshness and the snapshot worker, none of which is visible from outside. + +Neither is sufficient alone, and that is structural rather than a staffing +accident: each is blind to exactly what the other sees. `probe-ingest` stops at +the auth boundary and cannot tell a silent instance from a healthy one; +`probe-loopback` cannot report its own host being down, because with the runner +off its run queues instead of failing. See +[ADR-0026](../decisions/0026-scheduled-loopback-probe-with-a-pager.md). + +**Both are best-effort watchers, not monitoring.** GitHub schedules public repositories on a best-effort basis and *drops* ticks rather than delaying them — measured in this repository, one run out of the nine expected over nine hours — so **the detection delay is unbounded**, and a green workflow file is not -evidence that anything ran. It is strictly better than never looking, and that -is the whole of what it promises. See +evidence that anything ran. They are strictly better than never looking, and +that is the whole of what they promise. See [ADR-0025](../decisions/0025-best-effort-ingest-watcher-in-actions.md). The 10-minute systemd timer on the Pi that used to own this was **retired on @@ -18,8 +30,13 @@ The 10-minute systemd timer on the Pi that used to own this was **retired on with the production it watched, so it went down *with* cotel — which is the failure it existed to report. `~/ops/cotel-healthz.sh` is still on that machine and `--probe-only` still works by hand, but nothing ticks it, and nothing else -in `~/ops` watches cotel either. Any scheduled observation of cotel has to come -from outside that network, which is what this job is. +in `~/ops` watches cotel either. The board's order out of that retirement is +about the **vantage point**, not about `/healthz`: an observer that shares the +LAN, the uplink and the power outlet with production cannot report production +going down. `probe-ingest` satisfies it by running outside that network +entirely. `probe-loopback` satisfies it from the other side — it runs *on* the +deploy host, so it makes no claim about that host being reachable and leaves +that question to `probe-ingest`. A watcher of some kind is needed because a dead or silent cotel otherwise has no one looking at all: `Deploy` only runs on a push to `main`, the Cloudflare @@ -36,68 +53,74 @@ gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule ``` Empty output means no scheduled run has happened, however healthy the workflow -looks. That is the measurement of this watcher, and it is worth repeating -occasionally rather than assuming the first result holds. - -## What this watcher cannot see - -Three gaps, none of which it will report: - -- **A silent stale ingest behind a live tunnel.** The check stops at the auth - boundary, so an instance that answers 401 while accepting no spans, or whose - DuckDB is unreadable, is indistinguishable from health from outside. Only - `/healthz` carries that, and nothing asks `/healthz` on a schedule now. -- **A dead dashboard with a live ingest.** `cotel.aignite.pl` is a different - host behind Cloudflare Access and is not probed on any schedule. -- **Itself.** If the tick is dropped, or the runner cannot resolve the name, - nothing anywhere notices that the watcher is not watching. - -These gaps are not an oversight waiting to be closed quietly. They were weighed -on their own and the blindness is accepted for now, with its price written down: -see [ADR-0025](../decisions/0025-best-effort-ingest-watcher-in-actions.md). The -short version is that only `probe-loopback` can see the first two, scheduling it -without a pager would be the appearance of detection rather than detection, and -scheduling it *with* one is a decision about board-facing output rather than an -implementation detail. +looks. That is the measurement of both watchers, and it is worth repeating +occasionally rather than assuming the first result holds. Narrow it per job with +`--json name,conclusion,createdAt` and read which of the two actually ran: one +of them being green says nothing about the other having ticked. + +## What these watchers cannot see + +- **Itself.** If the tick is dropped, nothing anywhere notices that the watcher + is not watching. This is the gap that neither job closes for the other, and + the only remedy available is the measurement above. +- **The dashboard's path through Cloudflare.** `cotel.aignite.pl` is a different + host behind Cloudflare Access, and no scheduled job asks for it *through* + Access. A dead dashboard *process* is caught — `probe-loopback` asks + `/healthz` on the host — but a working process behind a broken edge is not. + See [the Access caveat](#the-access-caveat-the-dispatched-dashboard-edge-job). +- **A runner that cannot resolve the name.** Exit 7 is a defect in the observer, + not evidence about cotel; it warns and stays green rather than paging. + +Two gaps that used to be on this list — a silent stale ingest behind a live +tunnel, and a snapshot worker that has stopped producing restore points — were +closed on 2026-10-19 by putting `probe-loopback` on the schedule with a pager; +see [ADR-0026](../decisions/0026-scheduled-loopback-probe-with-a-pager.md). ## The vantage points Production is asked from several places, because each is blind to something the -others see. Only the first column is scheduled; the other two run on a -dispatch, as drills. +others see. The first two columns are scheduled; the third runs on a dispatch, +as a drill. -| | `probe-ingest` (**scheduled**) | `probe-loopback` (dispatch) | `probe-edge` (dispatch) | +| | `probe-ingest` (**scheduled**) | `probe-loopback` (**scheduled**) | `probe-edge` (dispatch) | |---|---|---|---| -| Scheduled | **yes, `*/15 * * * *`, best-effort** | no | no | -| Runner | `ubuntu-latest` — never self-hosted | `[self-hosted, flopsstuff, docker]` — the deploy host | `ubuntu-latest` | +| Scheduled | **yes, `*/15 * * * *`, best-effort** | **yes, the same `*/15`, best-effort** | no | +| Runner | `ubuntu-latest` — never self-hosted | `[self-hosted, flopsstuff, docker]` — the deploy host, never hosted | `ubuntu-latest` | | Target | `https://otlp.aignite.pl/v1/traces` (internet) | `http://127.0.0.1:8080/healthz` | `https://cotel.aignite.pl/healthz` | | Cloudflare in the path | yes (DNS + tunnel) | no | yes (tunnel + Access) | | Catches | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | process dead, crash loop, 503 DuckDB, stale/empty ingest, a snapshot worker that has stopped producing restore points | the dashboard host through Cloudflare | -| Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all | nothing in the path, but see the Access caveat below | +| Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all; and its own dropped ticks | nothing in the path, but see the Access caveat below | | Access service token | **not needed** | not needed | required | -| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | `[cotel-health-probe-drill]` — drill only | `[cotel-health-probe-drill-edge]` — drill only | - -`probe-ingest` runs on `ubuntu-latest` and must never be moved to the -self-hosted runner. An external vantage point is the entire reason this job -exists, and the self-hosted alternative is worse than it looks: a job on the -deploy host cannot report that host being off, because GitHub queues the run -rather than failing it, and `timeout-minutes` does **not** bound queue time — it -starts counting once a runner picks the job up. An absent host yields no colour -at all, and the only trace is a run sitting in `queued`, which is also what a -busy runner looks like. +| Ingest staleness window | n/a — never reads `/healthz` | **24 h** on a schedule, 6 h on a dispatch | 6 h | +| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-healthz-host]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | + +**Neither scheduled job may be moved to the other's runner.** `probe-ingest` +exists to hold an external vantage point, and the self-hosted alternative is +worse than it looks: a job on the deploy host cannot report that host being off, +because GitHub queues the run rather than failing it, and `timeout-minutes` does +**not** bound queue time — it starts counting once a runner picks the job up. An +absent host yields no colour at all, and the only trace is a run sitting in +`queued`, which is also what a busy runner looks like. `probe-loopback` is the +mirror image: `127.0.0.1:8080` and the derived `/api/v1/health` are reachable +from that host and nowhere else, so on a hosted runner it would have nothing to +ask. The pair is the design, not redundancy. Every prober uses a **different dedup marker** on purpose. On a single marker they would fight: a green tick on one would ask for the close of the alert -another had just raised. A dispatch is separated from the schedule for the -stronger version of the same reason — a drill must not touch a real alert; see +another had just raised — and the two halves genuinely go red independently, as +the 2026-10-14 tunnel outage showed, red from the internet while `/healthz` +answered 200 on the LAN throughout. Two markers open at once is two findings, +not a duplicate. A dispatch is separated from the schedule for the stronger +version of the same reason — a drill must not touch a real alert; see [Running a drill](#running-a-drill). -The `[cotel-health-probe]` marker, and the `/healthz` probe behind it, belong to -no scheduled caller any more. `scripts/probe-healthz.sh` is still here, still -tested, and still the right thing to run by hand or from a future scheduled -caller; what follows describes it on that basis. +The script's own default marker, `[cotel-health-probe]`, belonged to the retired +Pi timer and has no caller at all now. It is deliberately *not* reused by the +scheduled `/healthz` job: an alert may still be standing under it, and +inheriting one would make the new watcher start out deduped onto a ticket nobody +is working. -## What the `/healthz` probe checks (nothing runs it on a schedule) +## What the `/healthz` probe checks `scripts/probe-healthz.sh` classifies failure modes with distinct text, because they send the on-call looking in different places: @@ -114,9 +137,38 @@ they send the on-call looking in different places: A 200 with `ok: true` is not enough. Staleness is a body field (`newest_span_age_seconds`); the endpoint keeps 200 on a quiet instance so the -container HEALTHCHECK does not flap. The probe applies its own threshold: -**6 hours**. A present JSON `null` is an empty database (never ingested), which -is not the same as `0` (ingested just now) and not the same as a missing key. +container HEALTHCHECK does not flap. The probe applies its own threshold, +`STALE_AFTER_SECONDS`, defaulting to **6 hours** — but the scheduled job raises +it to **24**, and that number is measured rather than chosen: see +[the staleness threshold](#the-staleness-threshold-is-24-hours-on-the-schedule). +A present JSON `null` is an empty database (never ingested), which is not the +same as `0` (ingested just now) and not the same as a missing key. + +### The staleness threshold is 24 hours on the schedule + +Exit 3 — "ingest stale" — is a paging class, so its threshold decides how often +a quiet night mints a ticket. An hourly span histogram over one week of +production (`/api/v1/history?granularity=hour&range=week`) says 6 hours is not +an assertion about this instance: + +``` +gap 12h 2026-10-14 09:00 -> 21:00 (the tunnel outage — a real red) +gap 6h 2026-10-17 04:00 -> 10:00 +gap 6h 2026-10-15 22:00 -> 04:00 +gap 6h 2026-10-15 10:00 -> 16:00 +``` + +Ordinary silence here reaches six and seven hours overnight, so a scheduled +pager on the 6h window would have paged several times that week on nothing. A +full day with no span is an assertion — the longest silence ever measured on +this box, 12 hours, was the real tunnel outage. So the scheduled job sets +`STALE_AFTER_SECONDS=86400` and a dispatch keeps `21600`, where a human is +reading the answer and a tighter question is more useful. + +Moving this number is a decision about alert noise, and it needs the same +histogram taken again — not a guess. `SNAPSHOT_STALE_AFTER_SECONDS` is +unaffected either way: that worker ticks on its own timer and does not care how +much traffic arrives. If `last_ingest_at` / `newest_span_age_seconds` are absent, the probe degrades to liveness (HTTP 200) so it keeps working before that contract is on @@ -155,9 +207,10 @@ Exit **6** and not 5 because the retired `~/ops/cotel-healthz.sh` spends 5 on "the LAN half is green and the public ingest half is not", and the exit codes are a contract any future caller reads. -This verdict pages under the `/healthz` marker, but not in its words: the alert -is titled `cotel prod database snapshot is red` and leads with "the application -is alive; its backup is not". See [the two halves are classified +This verdict pages under the `/healthz` caller's marker — +`[cotel-healthz-host]` on the schedule — but not in its words: the alert is +titled `cotel prod database snapshot is red` and leads with "the application is +alive; its backup is not". See [the two halves are classified apart](#the-two-halves-are-classified-apart-and-page-apart). Two vantage points this must not be turned on for: anything reaching cotel @@ -211,9 +264,11 @@ Two deliberate narrownesses: check could not write anything. This probe is blind to what `/healthz` sees: it stops at the auth boundary and -says nothing about the database or ingest freshness. Both used to run on one -tick; this is the one that survives, so that blindness is now a standing gap — -see [What this watcher cannot see](#what-this-watcher-cannot-see). +says nothing about the database or ingest freshness. That is not a standing gap +any more — `probe-loopback` ticks on the same schedule and covers exactly it — +but it is still a reason never to read a green ingest tick as a verdict on the +instance. See [What these watchers cannot see](#what-these-watchers-cannot-see) +for what neither of them covers. ### How the scheduled job runs it @@ -269,8 +324,8 @@ that answers while the backup has stopped. So when the caller passes no | Verdict | Title | Lead says | |---|---|---| -| `no current database snapshot`, or `snapshots are required …` | `cotel prod database snapshot is red [cotel-health-probe]` | the application is alive, the backup is not; read the `snapshot` object in `/api/v1/health` and the worker's logs; do **not** restart cotel | -| anything else | `cotel prod /healthz is red [cotel-health-probe]` | the `/healthz` probe is red | +| `no current database snapshot`, or `snapshots are required …` | `cotel prod database snapshot is red []` | the application is alive, the backup is not; read the `snapshot` object in `/api/v1/health` and the worker's logs; do **not** restart cotel | +| anything else | `cotel prod /healthz is red []` | the `/healthz` probe is red | An explicit `PC_ALERT_SUBJECT`/`PC_ALERT_LEAD` always wins over the classification — that is how the ingest job keeps its own words. @@ -280,13 +335,21 @@ ticks of that probe — in either order, snapshot then `/healthz` or the reverse still dedup into one ticket. Were the subject part of the dedup key, one watcher's consecutive reds would mint an alert each. -One asymmetry the retired timer had and the scheduled job does not: while its -LAN half was red, its ingest half did not page, because a dead process makes -the public path unreachable as a consequence and a second alert would send its -reader hunting Cloudflare for a dead container. Nothing probes `/healthz` on a -schedule now, so there is no suppression to apply — which also means a red -ingest alert no longer implies anything about the process. Its description says -so, and tells the reader to check `/healthz` before touching the tunnel. +One asymmetry the retired timer had and neither scheduled job does: that timer +ran both halves in one process, so while its LAN half was red its ingest half +did not page — a dead process makes the public path unreachable as a consequence +and a second alert would send its reader hunting Cloudflare for a dead +container. **The two jobs here cannot suppress each other**, and not by +oversight: they run on different runners, keep no shared state, and their ticks +are dropped independently, so neither can know the other's verdict for this +minute. A dead process therefore pages twice, once per marker. + +Read that pair as one finding with two symptoms rather than two incidents. Which +one to trust about *what* is fixed: `[cotel-healthz-host]` is the one that sees +the process, so start there; the ingest alert's own description already tells its +reader to check `/healthz` before touching the tunnel. The reverse case is not +redundancy at all — an ingest alert alone, with `/healthz` green, is the +2026-10-14 tunnel shape and is exactly why the external half exists. ### The Access caveat (the dispatched dashboard-edge job) @@ -307,27 +370,29 @@ what would close one.** That was the earlier reading of it — that a meaningful check from outside the LAN had to wait for the token to be allowed on `cotel.aignite.pl`. It does not: the ingest host answers 401 to anyone, and that 401 proves DNS, tunnel, process and route, which is exactly what a probe on the -LAN cannot see. The scheduled job above uses it and asks for no credential. -What an Access-allowed token would add is narrower: the *dashboard* host's view -through Cloudflare, and the database and freshness fields that only `/healthz` -carries. Worth having, not worth blocking on. +LAN cannot see. The scheduled ingest job uses that and asks for no credential, +and the database and freshness fields that only `/healthz` carries are now +watched from the host itself. What an Access-allowed token would add is +narrower still: the *dashboard* host's view through Cloudflare. Worth having, +not worth blocking on. ## Schedule -`cron: "*/15 * * * *"` in `.github/workflows/health-probe.yml`, on the -`probe-ingest` job. Nothing else — not in this repo, not in `~/ops`, not in a -Paperclip routine — probes cotel on any schedule. +`cron: "*/15 * * * *"` in `.github/workflows/health-probe.yml`, driving both +`probe-ingest` and `probe-loopback`. Nothing else — not in this repo, not in +`~/ops`, not in a Paperclip routine — probes cotel on any schedule. **The interval is not the detection delay, and there is no bound on the detection delay.** GitHub drops scheduled ticks on public repositories, so the cadence set here is a ceiling on how often it *could* look, not a floor on how -often it does. 15 minutes is chosen because the cadence is free — a dropped -tick costs nothing and a kept one costs a single HTTP request — so the only -thing it trades against is nothing at all. +often it does. 15 minutes is chosen because the cadence is free — a dropped tick +costs nothing, and a kept one costs a couple of HTTP requests and a minute on a +runner we own — so the only thing it trades against is nothing at all. -What the schedule does bound is the alarm it can raise: one red tick is -confirmed by a second look 60 seconds later inside the same run, so a blinked -Cloudflare 5xx wakes nobody, and a confirmed red opens exactly one ticket. +What the schedule does bound is the alarm it can raise: in each job one red tick +is confirmed by a second look 60 seconds later inside the same run, so a blinked +Cloudflare 5xx or a container restarting through a deploy wakes nobody, and a +confirmed red opens exactly one ticket per marker. Verify the cadence rather than believing it: @@ -367,18 +432,31 @@ a red run against a dead endpoint; leave **page** unchecked unless you intend to open a Paperclip alert — see [Running a drill](#running-a-drill) for what a dispatch with **page** checked does. -A dispatch is the only way `probe-loopback` and `probe-edge` run at all, and it -is also how `probe-ingest` is rehearsed. The `test` job still runs on every pull -request and on pushes that touch the probe scripts, so the classification and -pager tests remain a merge gate. +A dispatch is the only way `probe-edge` runs at all; for the two scheduled jobs +it is the rehearsal path, and it pins drill markers so a rehearsal cannot touch +a real alert. The `test` job still runs on every pull request and on pushes that +touch the probe scripts, so the classification and pager tests remain a merge +gate. + +Because the dispatch inputs set a job's *whole* configuration, a dispatch is not +a faithful replay of a scheduled tick: on a dispatch `probe-loopback` keeps the +6h staleness window and the run id as its recovery key, where the schedule uses +24h and an hourly bucket. The pager path, the dedup pair and the in-run red +confirmation are identical, which is what a drill is testing. ## Running a drill -Every rehearsal pages a **drill** marker, never a production one. The production -marker `[cotel-ingest-edge]` belongs to `probe-ingest` **on a schedule only**; -the same job on a dispatch pins `[cotel-ingest-edge-drill]`, and the other two -jobs pin `[cotel-health-probe-drill]` and `[cotel-health-probe-drill-edge]` -unconditionally. `[cotel-health-probe]` has no caller at all now. +Every rehearsal pages a **drill** marker, never a production one. The two +production markers belong to a schedule only: `[cotel-ingest-edge]` to +`probe-ingest` and `[cotel-healthz-host]` to `probe-loopback`. Dispatched, the +same jobs pin `[cotel-ingest-edge-drill]` and `[cotel-health-probe-drill]`, and +`probe-edge` pins `[cotel-health-probe-drill-edge]` unconditionally. +`[cotel-health-probe]`, the script's default, has no caller at all. + +**Check the marker in the alert's title, not the job you dispatched.** A drill +that mints `[cotel-healthz-host]` means the conditional broke and the drill just +filed a production outage; that is the one failure mode in this section worth +looking for every time. Drill the scheduled watcher by dispatching **Health probe** three times, with **page** checked each time: @@ -433,14 +511,14 @@ because a coalesced wake is answered with the id of the run that absorbed it. Read `diagnostics/wakes` and match by `requestedAt`. A red dispatch also runs `probe-loopback` and `probe-edge`, which probe the real -endpoints on their own drill markers. Against a healthy production neither of -them pages, but if the self-hosted runner is offline `probe-loopback` queues and -the *run* never concludes — read `probe-ingest` on its own rather than the run's -overall conclusion. +endpoints on their own drill markers (an `ingest_url` override reaches neither of +them). Against a healthy production neither pages, but if the self-hosted runner +is offline `probe-loopback` queues and the *run* never concludes — read +`probe-ingest` on its own rather than the run's overall conclusion. The one thing to put back afterwards is the drill alert: it is a real issue -someone has to close. There is no shared failure streak left to reset — the -scheduled job keeps no state between runs. +someone has to close. There is no shared failure streak left to reset — neither +scheduled job keeps state between runs. Dedup is per marker, so the sets never see each other: a red drill cannot attach itself to a standing real alert, and — the half that actually bites — a @@ -458,6 +536,11 @@ assignment wakes its assignee), then once with the default (green — wakes that assignee again, who closes the alert from their own run). Never break production to get a red run. +The red leg now takes a minute longer than it used to: this job confirms a red +with a second probe 60 seconds later, so nothing pages until both looks fail. +The step log says so (`first probe exited N; looking again in 60s …`), and a +drill that pages immediately means the confirmation was skipped. + **Leave a gap between the red and green dispatches.** Wait for the alert's *assignment* run to finish before dispatching the green one. Back to back, the recovery wake is coalesced into the still-live assignment run, that run closes the alert, and the @@ -516,12 +599,12 @@ drill is over, and say in the close that they were drill artifacts. A red GitHub Actions run is **not** the page. Notifications on the Fl0p account are unproven (no `notifications` API scope, no public mailbox, agent identities are not GitHub users), and this company has already watched red Actions sit -unnoticed. The scheduled job going red is therefore not how anyone finds out; +unnoticed. A scheduled job going red is therefore not how anyone finds out; the Paperclip alert it opens is. On red, the prober opens a Paperclip issue titled `cotel is red -[]` — `public ingest at ` from the scheduled job, `prod /healthz` -by default — assigned to Daedalus. That assignment +[]` — `public ingest at ` from the ingest job, `prod /healthz` or +`prod database snapshot` from the loopback one — assigned to Daedalus. That assignment is the wake, and it is the only tracker issue the pager ever creates. A later tick searches `q=` and acts on the open issue whose **title** contains it. Search also matches comments and descriptions, so the first hit is not the @@ -602,9 +685,9 @@ rather than reporting an alert nobody was assigned. | `PC_API_TOKEN` | required | Agent API key. Create-only on issues | | `PC_COMPANY_ID` | required | Company the alert is opened in | | `PC_ASSIGNEE_AGENT_ID` | Daedalus | Assignee of a new alert. The assignment is the wake | -| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; the other two jobs pin `cotel-health-probe-drill{,-edge}`. Nothing uses the default | -| `PC_ALERT_MAX_AGE_H` | `6` | Hours an open alert stays a dedup target on a red tick. `0` always dedups. `probe-ingest` sets `0` | -| `PC_STILL_RED_WAKE` | `1` | Whether a red tick wakes the assignee of an already-open alert. `0` silences that wake and exits 0. `probe-ingest` sets `0` | +| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-healthz-host` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. Nothing uses the default, which belonged to the retired Pi timer | +| `PC_ALERT_MAX_AGE_H` | `6` | Hours an open alert stays a dedup target on a red tick. `0` always dedups. Both scheduled jobs set `0` | +| `PC_STILL_RED_WAKE` | `1` | Whether a red tick wakes the assignee of an already-open alert. `0` silences that wake and exits 0. Both scheduled jobs set `0` | | `PC_RUN_ID` | unset | Sent as `X-Paperclip-Run-Id` when set | | `CF_ACCESS_CLIENT_ID`, `CF_ACCESS_CLIENT_SECRET` | unset | Same pair as issue-sync. **In practice mandatory**: Cloudflare Access fronts the Paperclip API, and without them every call answers 302 and the pager reports `issue search: HTTP 302 … did not return JSON` | | `GITHUB_RUN_URL` | unset | Attached to the alert and the wake reason when set | @@ -612,16 +695,16 @@ rather than reporting an alert nobody was assigned. | `GITHUB_ACTOR` | unset | Quoted into a new alert's body when set | | `HEALTHZ_URL` | unset | Named in the alert so the assignee re-probes the same URL | | `GITHUB_RUN_ID` | unset | Fallback for `PC_CALL_ID` | -| `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. `probe-ingest` sends an hourly bucket (`ingest-`) rather than its run id: with the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another recovery wake | +| `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. Both scheduled jobs send an hourly bucket (`ingest-`, `healthz-host-`) rather than the run id: with the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another recovery wake. A dispatch keeps the run id — each drill is its own call | | `PC_SOURCE_LINE` | names Actions | First paragraph of a new alert: who opened it. Override it if you are not the workflow | -| `PC_REPROBE_HINT` | names an Actions dispatch | How the woken agent re-probes. `probe-ingest` replaces it with `bash scripts/probe-edge-ingest.sh`, which runs from any machine with internet | -| `PC_ALERT_SUBJECT` | classified from the verdict: `prod database snapshot` on a snapshot failure, else `prod /healthz` | What is red, in the title and in every wake reason. `probe-ingest` sets `public ingest at ` | +| `PC_REPROBE_HINT` | names a dispatch of `probe-loopback` | How the woken agent re-probes. Correct as-is only for `probe-loopback`'s own alerts; `probe-ingest` replaces it with `bash scripts/probe-edge-ingest.sh`, which runs from any machine with internet | +| `PC_ALERT_SUBJECT` | classified from the verdict: `prod database snapshot` on a snapshot failure, else `prod /healthz` | What is red, in the title and in every wake reason. Correct as-is for `probe-loopback`; `probe-ingest` sets `public ingest at ` | | `PC_ALERT_LEAD` | classified the same way | First line of a new alert's description. `probe-ingest` says spans are being dropped, that this check never reached `/healthz`, and that the watcher is best-effort | **`PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0` are a pair, and they put the whole burden on the recovery wake.** Together they give exactly one ticket per -outage and no further wakes while it stands, which is what the scheduled job is -asked for. The cost is that the open alert then dedups *every* later red tick +outage and no further wakes while it stands, which is what both scheduled jobs +are asked for. The cost is that the open alert then dedups *every* later red tick forever, so until it is closed the **next** outage wakes nobody. The recovery wake is the only thing that re-arms the pager — which is why the green path is not optional on a watcher that only has to detect failure, and why an alert @@ -688,12 +771,14 @@ nothing but that host can reach. The description names the URL that was probed so the agent re-checks the same endpoint. **Which re-probe depends on what paged.** The default text in -`page-cotel-health.sh` names a dispatch of the loopback half, which runs on the -host that may itself be the thing that is down and would then queue forever. -That is wrong for every caller, and the scheduled ingest job overrides it: its -alerts say to run `bash scripts/probe-edge-ingest.sh` from any machine with -internet, which needs no credential and no runner. Any new caller must override -it too — the defaults are not neutral. +`page-cotel-health.sh` names a dispatch of the loopback half. That is right for +exactly one caller — `probe-loopback`'s own alerts, where the host is +demonstrably up because it just ran the probe — and wrong for every other, since +that runner may *be* the dead host and a dispatch would then queue forever. The +scheduled ingest job therefore overrides it: its alerts say to run +`bash scripts/probe-edge-ingest.sh` from any machine with internet, which needs +no credential and no runner. Any new caller must decide this deliberately — the +defaults are not neutral, they are one caller's. That costs the woken agent one extra probe, by design. A live re-probe is better evidence than a payload minted ten minutes earlier, and it needs no passthrough @@ -727,10 +812,10 @@ Three more details matter in operation: heartbeat. A caller that ticks repeatedly while an alert stands wants neither extreme: the same key forever means one wake per alert, and a dropped wake then leaves a resolved alert standing, while a per-run key means a wake per - tick. So the scheduled job passes an hourly bucket - (`ingest-`). The still-red wake buckets on a 6h window — + tick. So each scheduled job passes an hourly bucket — `ingest-` + and `healthz-host-`. The still-red wake buckets on a 6h window — `cotel-health-still-red::` — for the callers that - leave it on; this one does not. + leave it on; neither of these does. - **An alert assigned to anyone else is a pager failure, by name.** An agent API key may wake only its own agent, so the pager reports `alert wake: HTTP 403, alert is assigned to an agent this credential @@ -757,15 +842,20 @@ step log, not the red X that the dead URL was always going to produce. The loopback job also checks for `jq` and `python3` on every run — that runner is a developer machine, not a managed image, so a missing interpreter surfaces -in a drill rather than during an incident. +on a green tick rather than during an incident. It is a `::warning`, so it does +not redden the job; grep the scheduled runs for it rather than waiting to be +told. Paperclip budget is spent only on a state change: the create on the first confirmed red, a create when the open alert has aged out of the dedup window -(not on the scheduled job, which has that window off), and a wake on a +(not on either scheduled job — both have that window off), and a wake on a recovery. A green tick with no open alert spends nothing — it costs one search. -A scheduled Paperclip routine that fires regardless of health was a rejected -alternative on cost (~24 full agent runs a day); see -[ADR-0025](../decisions/0025-best-effort-ingest-watcher-in-actions.md). +Two scheduled markers means the ceiling is two tickets at once rather than one, +and that ceiling is exactly what the dedup pair buys. A scheduled Paperclip +routine that fires regardless of health was a rejected alternative on cost (~24 +full agent runs a day); see +[ADR-0025](../decisions/0025-best-effort-ingest-watcher-in-actions.md) and +[ADR-0026](../decisions/0026-scheduled-loopback-probe-with-a-pager.md). ### Credentials @@ -784,9 +874,9 @@ fails to page, and *looks* like it paged — the pager's failure is a `issue search: HTTP 302` in the step log. That is why a drill's verdict is whether the page arrived, never the job's colour. -None of them had to be minted for this watcher: all five were already -repository secrets, and `probe-edge-ingest.sh` itself needs none at -all. A board API key — instance-admin authority over the whole tracker, handed +None of them had to be minted for either watcher: all five were already +repository secrets, and neither `probe-edge-ingest.sh` nor the loopback +`/healthz` probe needs any of them. A board API key — instance-admin authority over the whole tracker, handed to a public repository's CI — was refused in [ADR-0021](../decisions/0021-recovery-wakes-the-alerts-assignee.md), and that refusal stands. @@ -821,8 +911,10 @@ bash scripts/probe-edge-ingest_test.sh # wake and its off switch, per-call failure labels, caller overrides bash scripts/page-cotel-health_test.sh -# whether the scheduled watcher has actually been ticking +# whether the scheduled watchers have actually been ticking, and which of them gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule --limit 5 +gh run view --repo Flopsstuff/cotel --json jobs \ + --jq '.jobs[] | {name, conclusion}' ``` `probe-healthz.sh` exit codes: `0` healthy, `1` unreachable, `2` HTTP non-200, From 85810e49c5665670a7daebdcd5b767345f655ef0 Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 19 Oct 2026 14:16:52 +0200 Subject: [PATCH 2/6] fix(health): page the scheduled /healthz half under its historical marker MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The scheduled loopback job minted `[cotel-healthz-host]`, a marker invented for it on the grounds that the pager script's default — `[cotel-health-probe]`, this half's own historical marker — belonged to the retired Pi timer and might still have an alert standing under it. That hazard was measured rather than assumed: every issue carrying the exact `[cotel-health-probe]` marker is closed, and `find_open` only ever treats an open issue as a dedup target, so there is nothing to inherit. The marker the grant names is the one used. What the marker still must not be is the ingest half's `[cotel-ingest-edge]`: on a shared marker a green tick from one half asks for the close of an alert the other just raised. The drill latch on `workflow_dispatch` is unchanged. The recovery-key bucket follows the marker (`health-probe-`), and the passages that argued for a fresh marker now record the precondition for reusing this one instead. ADR-0025's consequences also name the grant's four conditions and the three states that stay unwatched after it: the dashboard host behind Access, a deploy host that is off producing no colour rather than red, and dropped ticks. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- .github/workflows/health-probe.yml | 10 +++---- ...5-best-effort-ingest-watcher-in-actions.md | 24 +++++++++++++++ ...6-scheduled-loopback-probe-with-a-pager.md | 18 ++++++----- docs/operations/duckdb-snapshots.md | 2 +- docs/operations/health-probe.md | 30 ++++++++++--------- 5 files changed, 56 insertions(+), 28 deletions(-) diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index a8e7124..ad0fbad 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -219,10 +219,10 @@ jobs: PC_COMPANY_ID: ${{ secrets.PAPERCLIP_COMPANY_ID }} # A dispatched run is an exercise and gets its own dedup marker, so a # drill can neither mint an alert that reads like an outage nor act on a - # standing real one. The scheduled marker is new rather than the script's - # default: the default belonged to the retired Pi timer and may still have - # an alert standing under it, which this job must not inherit. - PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-healthz-host' || 'cotel-health-probe-drill' }} + # standing real one. The scheduled marker is this half's historical one, + # distinct from the ingest half's: on a shared marker a green tick from + # one half would ask for the close of an alert the other had just raised. + PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-health-probe' || 'cotel-health-probe-drill' }} # These two are a pair. `0` switches the staleness window off, so an open # alert is always the dedup target and one outage mints exactly one # ticket; `PC_STILL_RED_WAKE=0` drops the per-tick wake that would @@ -242,7 +242,7 @@ jobs: # call and wants its own recovery. - name: Bucket the recovery key for the scheduled run if: github.event_name == 'schedule' - run: echo "PC_CALL_ID=healthz-host-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" + run: echo "PC_CALL_ID=health-probe-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" # The pager is the only thing here that wakes anyone, and this runner is # a developer Mac rather than a managed image. Check its interpreters on diff --git a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md index 132ebbd..7bda555 100644 --- a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md +++ b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md @@ -137,6 +137,30 @@ bound, and the first person to rely on that is the one holding a silent outage. on the same schedule with one. Everywhere this record says "nothing asks `/healthz` on a schedule now" — the bullet above and the stale-ingest bullet before it — read ADR-0026 instead. The rest of this record stands as written. + + The grant came with four conditions, and they are acceptance criteria rather + than advice. **One**, both pager steps fire on `inputs.page || github.event_name + == 'schedule'`: `schedule` supplies no inputs, so moving only the job's `if:` + would have produced a watcher that runs, goes red in Actions, and pages + nobody. **Two**, `PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`, the same + pair the ingest half carries — without them one outage mints up to 96 tickets + a day at `*/15`. **Three**, a production marker of its own, distinct from + `[cotel-ingest-edge]`, so that neither half's green tick closes the other + half's alert; the dispatch keeps paging a drill marker. **Four**, the snapshot + alert's wording is left alone — `scripts/page-cotel-health.sh` already titles + a snapshot failure `prod database snapshot` and leads with "the application is + alive; its backup is not". + + Three things stay unwatched afterwards. The **dashboard host through + Cloudflare Access** — a working process behind a broken edge still reads + green, and the request for an Access token to close that was withdrawn, so + `probe-edge` remains a dispatch-only drill. **A deploy host that is off + produces no colour at all** rather than red: the run sits in the queue, and + `timeout-minutes` does not bound queue time. Measured on 2026-10-19, deploy + `3a6997f7` waited in the queue from 05:26Z to 11:27Z, six hours. That case is + what the external ingest half is for. And **GitHub still drops scheduled + ticks**, for this half exactly as for the other, so the detection delay stays + unbounded and "did it run" stays a measurement. - **Watching the worker is only half of watching the restore point, and the other half is not a watcher question.** What a lost snapshot worker actually costs is bounded by what the off-host copy holds, and that copy decays on its diff --git a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md index 8f9f429..b740060 100644 --- a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md +++ b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md @@ -45,12 +45,14 @@ ticket, an in-run red confirmation, and a measured staleness threshold.** Six things had to be true together; the first five copy `probe-ingest`, whose shape the board accepted in the same form. -1. **Own marker: `[cotel-healthz-host]`.** New rather than reused. It must not - collide with `[cotel-ingest-edge]` — on one marker the two halves would - fight, a green tick on one asking for the close of an alert the other had - just raised — nor with the drill markers, nor with the script's default - `[cotel-health-probe]`, which belonged to the retired Pi timer and may still - have an alert standing under it that this job must not inherit. +1. **Own marker: `[cotel-health-probe]`**, this half's historical one and the + pager script's default. What it must not be is `[cotel-ingest-edge]` — on one + marker the two halves would fight, a green tick on one asking for the close + of an alert the other had just raised — nor either drill marker. Reusing the + retired Pi timer's marker carries one precondition, that nothing is left + standing under it: an inherited open alert would dedup the first real outage + onto a ticket nobody is working. Every alert under it was closed before the + schedule was turned on, and only an *open* issue is ever a dedup target. 2. **`PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`.** The staleness window off means an open alert is always the dedup target, so one outage mints exactly one ticket; the second drops the per-tick wake. Without the pair, @@ -60,7 +62,7 @@ shape the board accepted in the same form. later red tick, so until it is closed the *next* outage wakes nobody. ADR-0025 records "we only need failure, so drop the green path" as the tempting mistake, and it silently disarms the watcher. -4. **A coarse hourly recovery key, `healthz-host-`.** `PC_CALL_ID` +4. **A coarse hourly recovery key, `health-probe-`.** `PC_CALL_ID` feeds the recovery wake's idempotency key and defaults to the run id. With the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another wake — 96 @@ -123,7 +125,7 @@ is independent of the instance's traffic shape. `gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule` and empty output is the answer rather than an error. - **A second production marker exists, and the two must stay separate.** - `[cotel-healthz-host]` and `[cotel-ingest-edge]` watch different things from + `[cotel-health-probe]` and `[cotel-ingest-edge]` watch different things from different places and can be red independently: the tunnel outage of 2026-10-14 was red on the ingest marker while `/healthz` answered 200 on the LAN throughout. Merging them would make each half's green tick close the other diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 9a5edc8..3ce3fee 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -81,7 +81,7 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ **The `probe-loopback` job in `.github/workflows/health-probe.yml`, every 15 minutes, best-effort, since 2026-10-19.** `scripts/probe-healthz.sh` reads `/healthz` first and, when that is green, asks `/api/v1/health` and classifies -the `snapshot` object — a red verdict pages under the `[cotel-healthz-host]` +the `snapshot` object — a red verdict pages under the `[cotel-health-probe]` marker on a schedule, as an issue whose assignment wakes an agent. It runs on the self-hosted runner on the deploy host, which is the only vantage point from which `/api/v1/health` is reachable at all. The same probe still works by hand diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index bff9806..8bd8f1e 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -92,7 +92,7 @@ as a drill. | Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all; and its own dropped ticks | nothing in the path, but see the Access caveat below | | Access service token | **not needed** | not needed | required | | Ingest staleness window | n/a — never reads `/healthz` | **24 h** on a schedule, 6 h on a dispatch | 6 h | -| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-healthz-host]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | +| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-health-probe]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | **Neither scheduled job may be moved to the other's runner.** `probe-ingest` exists to hold an external vantage point, and the self-hosted alternative is @@ -114,11 +114,14 @@ not a duplicate. A dispatch is separated from the schedule for the stronger version of the same reason — a drill must not touch a real alert; see [Running a drill](#running-a-drill). -The script's own default marker, `[cotel-health-probe]`, belonged to the retired -Pi timer and has no caller at all now. It is deliberately *not* reused by the -scheduled `/healthz` job: an alert may still be standing under it, and -inheriting one would make the new watcher start out deduped onto a ticket nobody -is working. +The scheduled `/healthz` job pages under `[cotel-health-probe]`, which is both +this half's historical marker and the script's own default. The retired Pi timer +used it before, so the thing to check before trusting it again is that no alert +is left standing under it: an inherited one would dedup the first real outage +onto a ticket nobody is working. Every alert under that marker was closed when +the schedule was turned on; `scripts/page-cotel-health.sh` only ever treats an +*open* issue carrying the exact `[marker]` as a dedup target, so a closed one +costs nothing. ## What the `/healthz` probe checks @@ -208,7 +211,7 @@ Exit **6** and not 5 because the retired `~/ops/cotel-healthz.sh` spends 5 on are a contract any future caller reads. This verdict pages under the `/healthz` caller's marker — -`[cotel-healthz-host]` on the schedule — but not in its words: the alert is +`[cotel-health-probe]` on the schedule — but not in its words: the alert is titled `cotel prod database snapshot is red` and leads with "the application is alive; its backup is not". See [the two halves are classified apart](#the-two-halves-are-classified-apart-and-page-apart). @@ -345,7 +348,7 @@ are dropped independently, so neither can know the other's verdict for this minute. A dead process therefore pages twice, once per marker. Read that pair as one finding with two symptoms rather than two incidents. Which -one to trust about *what* is fixed: `[cotel-healthz-host]` is the one that sees +one to trust about *what* is fixed: `[cotel-health-probe]` is the one that sees the process, so start there; the ingest alert's own description already tells its reader to check `/healthz` before touching the tunnel. The reverse case is not redundancy at all — an ingest alert alone, with `/healthz` green, is the @@ -448,13 +451,12 @@ confirmation are identical, which is what a drill is testing. Every rehearsal pages a **drill** marker, never a production one. The two production markers belong to a schedule only: `[cotel-ingest-edge]` to -`probe-ingest` and `[cotel-healthz-host]` to `probe-loopback`. Dispatched, the +`probe-ingest` and `[cotel-health-probe]` to `probe-loopback`. Dispatched, the same jobs pin `[cotel-ingest-edge-drill]` and `[cotel-health-probe-drill]`, and `probe-edge` pins `[cotel-health-probe-drill-edge]` unconditionally. -`[cotel-health-probe]`, the script's default, has no caller at all. **Check the marker in the alert's title, not the job you dispatched.** A drill -that mints `[cotel-healthz-host]` means the conditional broke and the drill just +that mints `[cotel-health-probe]` means the conditional broke and the drill just filed a production outage; that is the one failure mode in this section worth looking for every time. @@ -685,7 +687,7 @@ rather than reporting an alert nobody was assigned. | `PC_API_TOKEN` | required | Agent API key. Create-only on issues | | `PC_COMPANY_ID` | required | Company the alert is opened in | | `PC_ASSIGNEE_AGENT_ID` | Daedalus | Assignee of a new alert. The assignment is the wake | -| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-healthz-host` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. Nothing uses the default, which belonged to the retired Pi timer | +| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-health-probe` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. The scheduled `/healthz` job is the default's only caller | | `PC_ALERT_MAX_AGE_H` | `6` | Hours an open alert stays a dedup target on a red tick. `0` always dedups. Both scheduled jobs set `0` | | `PC_STILL_RED_WAKE` | `1` | Whether a red tick wakes the assignee of an already-open alert. `0` silences that wake and exits 0. Both scheduled jobs set `0` | | `PC_RUN_ID` | unset | Sent as `X-Paperclip-Run-Id` when set | @@ -695,7 +697,7 @@ rather than reporting an alert nobody was assigned. | `GITHUB_ACTOR` | unset | Quoted into a new alert's body when set | | `HEALTHZ_URL` | unset | Named in the alert so the assignee re-probes the same URL | | `GITHUB_RUN_ID` | unset | Fallback for `PC_CALL_ID` | -| `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. Both scheduled jobs send an hourly bucket (`ingest-`, `healthz-host-`) rather than the run id: with the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another recovery wake. A dispatch keeps the run id — each drill is its own call | +| `PC_CALL_ID` | `GITHUB_RUN_ID`, then `manual` | Unique half of the recovery idempotency key. Both scheduled jobs send an hourly bucket (`ingest-`, `health-probe-`) rather than the run id: with the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another recovery wake. A dispatch keeps the run id — each drill is its own call | | `PC_SOURCE_LINE` | names Actions | First paragraph of a new alert: who opened it. Override it if you are not the workflow | | `PC_REPROBE_HINT` | names a dispatch of `probe-loopback` | How the woken agent re-probes. Correct as-is only for `probe-loopback`'s own alerts; `probe-ingest` replaces it with `bash scripts/probe-edge-ingest.sh`, which runs from any machine with internet | | `PC_ALERT_SUBJECT` | classified from the verdict: `prod database snapshot` on a snapshot failure, else `prod /healthz` | What is red, in the title and in every wake reason. Correct as-is for `probe-loopback`; `probe-ingest` sets `public ingest at ` | @@ -813,7 +815,7 @@ Three more details matter in operation: extreme: the same key forever means one wake per alert, and a dropped wake then leaves a resolved alert standing, while a per-run key means a wake per tick. So each scheduled job passes an hourly bucket — `ingest-` - and `healthz-host-`. The still-red wake buckets on a 6h window — + and `health-probe-`. The still-red wake buckets on a 6h window — `cotel-health-still-red::` — for the callers that leave it on; neither of these does. - **An alert assigned to anyone else is a pager failure, by name.** An agent From 60be1582720cf32eff50edc56d6af8b7b1ff9921 Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 19 Oct 2026 14:19:29 +0200 Subject: [PATCH 3/6] fix(health): restore the separate scheduled marker the grant asked for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit moved the scheduled `/healthz` marker to the pager script's default, `[cotel-health-probe]`, on the grounds that nothing is standing under it. The standing-alert check was sound; the conclusion was not what was granted. The CEO's conditions name "a *separate* `PC_ORIGIN_ID` for the scheduled marker", and the CTO's proposal that those conditions accepted names `cotel-healthz-host` explicitly as a new name rather than the script default. So the marker and its recovery bucket go back to `cotel-healthz-host` / `healthz-host-`. Beyond the grant, there is a second reason the default is the wrong slot for production, and it is not time-local the way "nothing is standing under it" is. The default is what *any* caller that forgets `PC_ORIGIN_ID` writes into — a by-hand `page-cotel-health.sh`, or `~/ops/cotel-healthz.sh`, which is still on the Pi and still runs by hand. Production's dedup slot should not be reachable by forgetting a variable, and "nothing uses the default" is a property worth keeping true. Both reasons are now recorded where the choice is made. The four-conditions and remaining-blindness passages added to ADR-0025 by the previous commit are unaffected and kept as written. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- .github/workflows/health-probe.yml | 11 +++--- ...6-scheduled-loopback-probe-with-a-pager.md | 25 +++++++------ docs/operations/duckdb-snapshots.md | 2 +- docs/operations/health-probe.md | 35 ++++++++++--------- 4 files changed, 40 insertions(+), 33 deletions(-) diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index ad0fbad..abee3c9 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -219,10 +219,11 @@ jobs: PC_COMPANY_ID: ${{ secrets.PAPERCLIP_COMPANY_ID }} # A dispatched run is an exercise and gets its own dedup marker, so a # drill can neither mint an alert that reads like an outage nor act on a - # standing real one. The scheduled marker is this half's historical one, - # distinct from the ingest half's: on a shared marker a green tick from - # one half would ask for the close of an alert the other had just raised. - PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-health-probe' || 'cotel-health-probe-drill' }} + # standing real one. The scheduled marker is new rather than the script's + # default: that default belonged to the retired Pi timer, and it is also + # what any caller which forgets to set this variable writes into — so + # production's dedup slot must not be reachable by forgetting. + PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-healthz-host' || 'cotel-health-probe-drill' }} # These two are a pair. `0` switches the staleness window off, so an open # alert is always the dedup target and one outage mints exactly one # ticket; `PC_STILL_RED_WAKE=0` drops the per-tick wake that would @@ -242,7 +243,7 @@ jobs: # call and wants its own recovery. - name: Bucket the recovery key for the scheduled run if: github.event_name == 'schedule' - run: echo "PC_CALL_ID=health-probe-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" + run: echo "PC_CALL_ID=healthz-host-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" # The pager is the only thing here that wakes anyone, and this runner is # a developer Mac rather than a managed image. Check its interpreters on diff --git a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md index b740060..005ba10 100644 --- a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md +++ b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md @@ -45,14 +45,19 @@ ticket, an in-run red confirmation, and a measured staleness threshold.** Six things had to be true together; the first five copy `probe-ingest`, whose shape the board accepted in the same form. -1. **Own marker: `[cotel-health-probe]`**, this half's historical one and the - pager script's default. What it must not be is `[cotel-ingest-edge]` — on one - marker the two halves would fight, a green tick on one asking for the close - of an alert the other had just raised — nor either drill marker. Reusing the - retired Pi timer's marker carries one precondition, that nothing is left - standing under it: an inherited open alert would dedup the first real outage - onto a ticket nobody is working. Every alert under it was closed before the - schedule was turned on, and only an *open* issue is ever a dedup target. +1. **Own marker: `[cotel-healthz-host]`.** New rather than reused, which is the + grant's own condition — a *separate* production marker. It must not be + `[cotel-ingest-edge]`: on one marker the two halves would fight, a green tick + on one asking for the close of an alert the other had just raised. Nor either + drill marker. And deliberately not the pager script's default, + `[cotel-health-probe]`, for two reasons. That default belonged to the retired + Pi timer, so an alert may be standing under it, and an inherited open alert + would dedup the first real outage onto a ticket nobody is working. More + durably: the default is what *any* caller which forgets to set `PC_ORIGIN_ID` + writes into — a by-hand `page-cotel-health.sh`, or `~/ops/cotel-healthz.sh`, + which is still on the Pi and still runs by hand. Production's dedup slot must + not be reachable by forgetting. "Nothing uses the default" is a property + worth keeping true. 2. **`PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`.** The staleness window off means an open alert is always the dedup target, so one outage mints exactly one ticket; the second drops the per-tick wake. Without the pair, @@ -62,7 +67,7 @@ shape the board accepted in the same form. later red tick, so until it is closed the *next* outage wakes nobody. ADR-0025 records "we only need failure, so drop the green path" as the tempting mistake, and it silently disarms the watcher. -4. **A coarse hourly recovery key, `health-probe-`.** `PC_CALL_ID` +4. **A coarse hourly recovery key, `healthz-host-`.** `PC_CALL_ID` feeds the recovery wake's idempotency key and defaults to the run id. With the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another wake — 96 @@ -125,7 +130,7 @@ is independent of the instance's traffic shape. `gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule` and empty output is the answer rather than an error. - **A second production marker exists, and the two must stay separate.** - `[cotel-health-probe]` and `[cotel-ingest-edge]` watch different things from + `[cotel-healthz-host]` and `[cotel-ingest-edge]` watch different things from different places and can be red independently: the tunnel outage of 2026-10-14 was red on the ingest marker while `/healthz` answered 200 on the LAN throughout. Merging them would make each half's green tick close the other diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 3ce3fee..9a5edc8 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -81,7 +81,7 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ **The `probe-loopback` job in `.github/workflows/health-probe.yml`, every 15 minutes, best-effort, since 2026-10-19.** `scripts/probe-healthz.sh` reads `/healthz` first and, when that is green, asks `/api/v1/health` and classifies -the `snapshot` object — a red verdict pages under the `[cotel-health-probe]` +the `snapshot` object — a red verdict pages under the `[cotel-healthz-host]` marker on a schedule, as an issue whose assignment wakes an agent. It runs on the self-hosted runner on the deploy host, which is the only vantage point from which `/api/v1/health` is reachable at all. The same probe still works by hand diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 8bd8f1e..c25e0c0 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -92,7 +92,7 @@ as a drill. | Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all; and its own dropped ticks | nothing in the path, but see the Access caveat below | | Access service token | **not needed** | not needed | required | | Ingest staleness window | n/a — never reads `/healthz` | **24 h** on a schedule, 6 h on a dispatch | 6 h | -| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-health-probe]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | +| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-healthz-host]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | **Neither scheduled job may be moved to the other's runner.** `probe-ingest` exists to hold an external vantage point, and the self-hosted alternative is @@ -114,14 +114,14 @@ not a duplicate. A dispatch is separated from the schedule for the stronger version of the same reason — a drill must not touch a real alert; see [Running a drill](#running-a-drill). -The scheduled `/healthz` job pages under `[cotel-health-probe]`, which is both -this half's historical marker and the script's own default. The retired Pi timer -used it before, so the thing to check before trusting it again is that no alert -is left standing under it: an inherited one would dedup the first real outage -onto a ticket nobody is working. Every alert under that marker was closed when -the schedule was turned on; `scripts/page-cotel-health.sh` only ever treats an -*open* issue carrying the exact `[marker]` as a dedup target, so a closed one -costs nothing. +The script's own default marker, `[cotel-health-probe]`, belonged to the retired +Pi timer and has no caller at all. The scheduled `/healthz` job deliberately does +*not* reuse it, for two reasons: an alert may still be standing under it, and +inheriting one would make the new watcher start out deduped onto a ticket nobody +is working; and the default is what any caller that forgets `PC_ORIGIN_ID` +writes into — a by-hand `page-cotel-health.sh`, or `~/ops/cotel-healthz.sh`, +which is still on the Pi. Keep "nothing uses the default" true, and forgetting +the variable can never land in production's dedup slot. ## What the `/healthz` probe checks @@ -211,7 +211,7 @@ Exit **6** and not 5 because the retired `~/ops/cotel-healthz.sh` spends 5 on are a contract any future caller reads. This verdict pages under the `/healthz` caller's marker — -`[cotel-health-probe]` on the schedule — but not in its words: the alert is +`[cotel-healthz-host]` on the schedule — but not in its words: the alert is titled `cotel prod database snapshot is red` and leads with "the application is alive; its backup is not". See [the two halves are classified apart](#the-two-halves-are-classified-apart-and-page-apart). @@ -298,13 +298,13 @@ for what neither of them covers. "The process is dead" and "the process is fine, the public path is down" need different people doing different things, so they never share an alert: -- **separate dedup markers** — `[cotel-health-probe]` and `[cotel-ingest-edge]`. +- **separate dedup markers** — `[cotel-healthz-host]` and `[cotel-ingest-edge]`. On one marker the halves would fight, a green tick of one asking for the close of the alert the other just raised; - **separate failure streaks** — the retired timer kept them in `reports/cotel-healthz/state` and `state-edge`, so an uplink blinking could - not page for a healthy application and vice versa. The scheduled job has no - store between runs and confirms a red inside one run instead; + not page for a healthy application and vice versa. Neither scheduled job has a + store between runs; each confirms a red inside one run instead; - **different words** — the ingest alert's title, first line and wake reason say that the application is alive and its public ingest path is not, and tell the reader to look at the tunnel and DNS rather than restart the container. The @@ -348,7 +348,7 @@ are dropped independently, so neither can know the other's verdict for this minute. A dead process therefore pages twice, once per marker. Read that pair as one finding with two symptoms rather than two incidents. Which -one to trust about *what* is fixed: `[cotel-health-probe]` is the one that sees +one to trust about *what* is fixed: `[cotel-healthz-host]` is the one that sees the process, so start there; the ingest alert's own description already tells its reader to check `/healthz` before touching the tunnel. The reverse case is not redundancy at all — an ingest alert alone, with `/healthz` green, is the @@ -451,12 +451,13 @@ confirmation are identical, which is what a drill is testing. Every rehearsal pages a **drill** marker, never a production one. The two production markers belong to a schedule only: `[cotel-ingest-edge]` to -`probe-ingest` and `[cotel-health-probe]` to `probe-loopback`. Dispatched, the +`probe-ingest` and `[cotel-healthz-host]` to `probe-loopback`. Dispatched, the same jobs pin `[cotel-ingest-edge-drill]` and `[cotel-health-probe-drill]`, and `probe-edge` pins `[cotel-health-probe-drill-edge]` unconditionally. +`[cotel-health-probe]`, the script's default, has no caller at all. **Check the marker in the alert's title, not the job you dispatched.** A drill -that mints `[cotel-health-probe]` means the conditional broke and the drill just +that mints `[cotel-healthz-host]` means the conditional broke and the drill just filed a production outage; that is the one failure mode in this section worth looking for every time. @@ -687,7 +688,7 @@ rather than reporting an alert nobody was assigned. | `PC_API_TOKEN` | required | Agent API key. Create-only on issues | | `PC_COMPANY_ID` | required | Company the alert is opened in | | `PC_ASSIGNEE_AGENT_ID` | Daedalus | Assignee of a new alert. The assignment is the wake | -| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-health-probe` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. The scheduled `/healthz` job is the default's only caller | +| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-healthz-host` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. Nothing uses the default, and nothing should: it is what a caller that forgets this variable writes into | | `PC_ALERT_MAX_AGE_H` | `6` | Hours an open alert stays a dedup target on a red tick. `0` always dedups. Both scheduled jobs set `0` | | `PC_STILL_RED_WAKE` | `1` | Whether a red tick wakes the assignee of an already-open alert. `0` silences that wake and exits 0. Both scheduled jobs set `0` | | `PC_RUN_ID` | unset | Sent as `X-Paperclip-Run-Id` when set | From 666230356557668a4b574595ff9b8adaeb2241fa Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 19 Oct 2026 14:22:32 +0200 Subject: [PATCH 4/6] fix(health): the marker names the subject, so the /healthz half shares it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restores the scheduled marker to `[cotel-health-probe]`, reverting the previous commit, and records the evidence that settles the choice rather than leaving it to be re-argued. The case for a fresh name was that the pager script's default is what any caller forgetting `PC_ORIGIN_ID` writes into, so production's dedup slot should not be reachable by forgetting. The premise is wrong: nothing reaches that slot by accident. `~/ops/cotel-healthz.sh` sets it deliberately — the LAN `/healthz` half under `cotel-health-probe` and the public ingest half under `cotel-ingest-edge`, the latter already shared with the scheduled `probe-ingest`. That sharing is the established shape, not an oversight. A marker names a subject, not a prober. Two probers asking the same question share a marker so that one outage stays one ticket whichever of them saw it first; a second name for the `/healthz` subject would let a single outage raise two tickets, which is what the dedup pair exists to prevent. The CEO's condition is that the production marker be separate from `cotel-ingest-edge` — a different subject — and this satisfies it. The standing-alert precondition still holds and still passes: every issue carrying the exact marker is closed, and only an open one is ever a dedup target. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- .github/workflows/health-probe.yml | 11 ++-- ...6-scheduled-loopback-probe-with-a-pager.md | 37 ++++++++----- docs/operations/duckdb-snapshots.md | 2 +- docs/operations/health-probe.md | 55 ++++++++++--------- 4 files changed, 58 insertions(+), 47 deletions(-) diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index abee3c9..ad0fbad 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -219,11 +219,10 @@ jobs: PC_COMPANY_ID: ${{ secrets.PAPERCLIP_COMPANY_ID }} # A dispatched run is an exercise and gets its own dedup marker, so a # drill can neither mint an alert that reads like an outage nor act on a - # standing real one. The scheduled marker is new rather than the script's - # default: that default belonged to the retired Pi timer, and it is also - # what any caller which forgets to set this variable writes into — so - # production's dedup slot must not be reachable by forgetting. - PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-healthz-host' || 'cotel-health-probe-drill' }} + # standing real one. The scheduled marker is this half's historical one, + # distinct from the ingest half's: on a shared marker a green tick from + # one half would ask for the close of an alert the other had just raised. + PC_ORIGIN_ID: ${{ github.event_name == 'schedule' && 'cotel-health-probe' || 'cotel-health-probe-drill' }} # These two are a pair. `0` switches the staleness window off, so an open # alert is always the dedup target and one outage mints exactly one # ticket; `PC_STILL_RED_WAKE=0` drops the per-tick wake that would @@ -243,7 +242,7 @@ jobs: # call and wants its own recovery. - name: Bucket the recovery key for the scheduled run if: github.event_name == 'schedule' - run: echo "PC_CALL_ID=healthz-host-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" + run: echo "PC_CALL_ID=health-probe-$(( $(date -u +%s) / 3600 ))" >> "$GITHUB_ENV" # The pager is the only thing here that wakes anyone, and this runner is # a developer Mac rather than a managed image. Check its interpreters on diff --git a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md index 005ba10..c51c902 100644 --- a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md +++ b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md @@ -45,19 +45,26 @@ ticket, an in-run red confirmation, and a measured staleness threshold.** Six things had to be true together; the first five copy `probe-ingest`, whose shape the board accepted in the same form. -1. **Own marker: `[cotel-healthz-host]`.** New rather than reused, which is the - grant's own condition — a *separate* production marker. It must not be - `[cotel-ingest-edge]`: on one marker the two halves would fight, a green tick - on one asking for the close of an alert the other had just raised. Nor either - drill marker. And deliberately not the pager script's default, - `[cotel-health-probe]`, for two reasons. That default belonged to the retired - Pi timer, so an alert may be standing under it, and an inherited open alert - would dedup the first real outage onto a ticket nobody is working. More - durably: the default is what *any* caller which forgets to set `PC_ORIGIN_ID` - writes into — a by-hand `page-cotel-health.sh`, or `~/ops/cotel-healthz.sh`, - which is still on the Pi and still runs by hand. Production's dedup slot must - not be reachable by forgetting. "Nothing uses the default" is a property - worth keeping true. +1. **Own marker: `[cotel-health-probe]`**, this half's historical one and the + pager script's default. What it must not be is `[cotel-ingest-edge]` — on one + marker the two halves would fight, a green tick on one asking for the close + of an alert the other had just raised — nor either drill marker. + + A marker names a **subject**, not a prober, and the convention already in + use settles which name this is. `~/ops/cotel-healthz.sh` on the Pi still + pages by hand, and it pages the LAN `/healthz` half under + `[cotel-health-probe]` and the public ingest half under + `[cotel-ingest-edge]` — the second of which the scheduled `probe-ingest` + already shares with it. Two probers on one subject sharing one marker is the + accepted shape, and it is what makes one outage one ticket no matter which + of them saw it first. Minting a *new* name for the scheduled `/healthz` job + would give one subject two markers and let a single outage raise two + tickets, which is the failure the dedup pair exists to prevent. + + Reusing it carries one precondition, that nothing is left standing under it: + an inherited open alert would dedup the first real outage onto a ticket + nobody is working. Every alert under it was closed before the schedule was + turned on, and only an *open* issue is ever a dedup target. 2. **`PC_ALERT_MAX_AGE_H=0` and `PC_STILL_RED_WAKE=0`.** The staleness window off means an open alert is always the dedup target, so one outage mints exactly one ticket; the second drops the per-tick wake. Without the pair, @@ -67,7 +74,7 @@ shape the board accepted in the same form. later red tick, so until it is closed the *next* outage wakes nobody. ADR-0025 records "we only need failure, so drop the green path" as the tempting mistake, and it silently disarms the watcher. -4. **A coarse hourly recovery key, `healthz-host-`.** `PC_CALL_ID` +4. **A coarse hourly recovery key, `health-probe-`.** `PC_CALL_ID` feeds the recovery wake's idempotency key and defaults to the run id. With the staleness window off an alert can stand for a long time, and a per-run key would make every green tick in the meantime mint another wake — 96 @@ -130,7 +137,7 @@ is independent of the instance's traffic shape. `gh run list --repo Flopsstuff/cotel --workflow=health-probe.yml --event=schedule` and empty output is the answer rather than an error. - **A second production marker exists, and the two must stay separate.** - `[cotel-healthz-host]` and `[cotel-ingest-edge]` watch different things from + `[cotel-health-probe]` and `[cotel-ingest-edge]` watch different things from different places and can be red independently: the tunnel outage of 2026-10-14 was red on the ingest marker while `/healthz` answered 200 on the LAN throughout. Merging them would make each half's green tick close the other diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 9a5edc8..3ce3fee 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -81,7 +81,7 @@ docker run --rm -v cotel-snapshots:/snapshots debian:bookworm-slim \ **The `probe-loopback` job in `.github/workflows/health-probe.yml`, every 15 minutes, best-effort, since 2026-10-19.** `scripts/probe-healthz.sh` reads `/healthz` first and, when that is green, asks `/api/v1/health` and classifies -the `snapshot` object — a red verdict pages under the `[cotel-healthz-host]` +the `snapshot` object — a red verdict pages under the `[cotel-health-probe]` marker on a schedule, as an issue whose assignment wakes an agent. It runs on the self-hosted runner on the deploy host, which is the only vantage point from which `/api/v1/health` is reachable at all. The same probe still works by hand diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index c25e0c0..00478b4 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -92,7 +92,7 @@ as a drill. | Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all; and its own dropped ticks | nothing in the path, but see the Access caveat below | | Access service token | **not needed** | not needed | required | | Ingest staleness window | n/a — never reads `/healthz` | **24 h** on a schedule, 6 h on a dispatch | 6 h | -| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-healthz-host]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | +| Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-health-probe]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | **Neither scheduled job may be moved to the other's runner.** `probe-ingest` exists to hold an external vantage point, and the self-hosted alternative is @@ -105,23 +105,29 @@ mirror image: `127.0.0.1:8080` and the derived `/api/v1/health` are reachable from that host and nowhere else, so on a hosted runner it would have nothing to ask. The pair is the design, not redundancy. -Every prober uses a **different dedup marker** on purpose. On a single marker -they would fight: a green tick on one would ask for the close of the alert -another had just raised — and the two halves genuinely go red independently, as -the 2026-10-14 tunnel outage showed, red from the internet while `/healthz` -answered 200 on the LAN throughout. Two markers open at once is two findings, -not a duplicate. A dispatch is separated from the schedule for the stronger -version of the same reason — a drill must not touch a real alert; see +**A marker names a subject, not a prober.** The two scheduled halves ask about +different things, so they page different markers: on a single one they would +fight, a green tick from one asking for the close of the alert the other had +just raised — and they genuinely go red independently, as the 2026-10-14 tunnel +outage showed, red from the internet while `/healthz` answered 200 on the LAN +throughout. Both markers open at once is two findings, not a duplicate. A +dispatch is separated from the schedule for the stronger version of the same +reason — a drill must not touch a real alert; see [Running a drill](#running-a-drill). -The script's own default marker, `[cotel-health-probe]`, belonged to the retired -Pi timer and has no caller at all. The scheduled `/healthz` job deliberately does -*not* reuse it, for two reasons: an alert may still be standing under it, and -inheriting one would make the new watcher start out deduped onto a ticket nobody -is working; and the default is what any caller that forgets `PC_ORIGIN_ID` -writes into — a by-hand `page-cotel-health.sh`, or `~/ops/cotel-healthz.sh`, -which is still on the Pi. Keep "nothing uses the default" true, and forgetting -the variable can never land in production's dedup slot. +Two probers on the *same* subject share its marker, which is why the scheduled +`/healthz` job pages under `[cotel-health-probe]` — this half's historical +marker and `page-cotel-health.sh`'s default. `~/ops/cotel-healthz.sh` on the Pi +still pages by hand under that same marker, exactly as it shares +`[cotel-ingest-edge]` with the scheduled `probe-ingest`. Sharing is the point: +one outage stays one ticket no matter which prober saw it first, where a second +name for the same subject would let one outage raise two. + +The one precondition for reusing it is that nothing is standing under it: an +inherited open alert would dedup the first real outage onto a ticket nobody is +working. Every alert under that marker was closed when the schedule was turned +on; `scripts/page-cotel-health.sh` only ever treats an *open* issue carrying the +exact `[marker]` as a dedup target, so a closed one costs nothing. ## What the `/healthz` probe checks @@ -211,7 +217,7 @@ Exit **6** and not 5 because the retired `~/ops/cotel-healthz.sh` spends 5 on are a contract any future caller reads. This verdict pages under the `/healthz` caller's marker — -`[cotel-healthz-host]` on the schedule — but not in its words: the alert is +`[cotel-health-probe]` on the schedule — but not in its words: the alert is titled `cotel prod database snapshot is red` and leads with "the application is alive; its backup is not". See [the two halves are classified apart](#the-two-halves-are-classified-apart-and-page-apart). @@ -298,13 +304,13 @@ for what neither of them covers. "The process is dead" and "the process is fine, the public path is down" need different people doing different things, so they never share an alert: -- **separate dedup markers** — `[cotel-healthz-host]` and `[cotel-ingest-edge]`. +- **separate dedup markers** — `[cotel-health-probe]` and `[cotel-ingest-edge]`. On one marker the halves would fight, a green tick of one asking for the close of the alert the other just raised; - **separate failure streaks** — the retired timer kept them in `reports/cotel-healthz/state` and `state-edge`, so an uplink blinking could - not page for a healthy application and vice versa. Neither scheduled job has a - store between runs; each confirms a red inside one run instead; + not page for a healthy application and vice versa. The scheduled job has no + store between runs and confirms a red inside one run instead; - **different words** — the ingest alert's title, first line and wake reason say that the application is alive and its public ingest path is not, and tell the reader to look at the tunnel and DNS rather than restart the container. The @@ -348,7 +354,7 @@ are dropped independently, so neither can know the other's verdict for this minute. A dead process therefore pages twice, once per marker. Read that pair as one finding with two symptoms rather than two incidents. Which -one to trust about *what* is fixed: `[cotel-healthz-host]` is the one that sees +one to trust about *what* is fixed: `[cotel-health-probe]` is the one that sees the process, so start there; the ingest alert's own description already tells its reader to check `/healthz` before touching the tunnel. The reverse case is not redundancy at all — an ingest alert alone, with `/healthz` green, is the @@ -451,13 +457,12 @@ confirmation are identical, which is what a drill is testing. Every rehearsal pages a **drill** marker, never a production one. The two production markers belong to a schedule only: `[cotel-ingest-edge]` to -`probe-ingest` and `[cotel-healthz-host]` to `probe-loopback`. Dispatched, the +`probe-ingest` and `[cotel-health-probe]` to `probe-loopback`. Dispatched, the same jobs pin `[cotel-ingest-edge-drill]` and `[cotel-health-probe-drill]`, and `probe-edge` pins `[cotel-health-probe-drill-edge]` unconditionally. -`[cotel-health-probe]`, the script's default, has no caller at all. **Check the marker in the alert's title, not the job you dispatched.** A drill -that mints `[cotel-healthz-host]` means the conditional broke and the drill just +that mints `[cotel-health-probe]` means the conditional broke and the drill just filed a production outage; that is the one failure mode in this section worth looking for every time. @@ -688,7 +693,7 @@ rather than reporting an alert nobody was assigned. | `PC_API_TOKEN` | required | Agent API key. Create-only on issues | | `PC_COMPANY_ID` | required | Company the alert is opened in | | `PC_ASSIGNEE_AGENT_ID` | Daedalus | Assignee of a new alert. The assignment is the wake | -| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-healthz-host` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. Nothing uses the default, and nothing should: it is what a caller that forgets this variable writes into | +| `PC_ORIGIN_ID` | `cotel-health-probe` | Dedup marker. `probe-ingest` sets `cotel-ingest-edge` on a schedule and `cotel-ingest-edge-drill` on a dispatch; `probe-loopback` sets `cotel-health-probe` on a schedule and `cotel-health-probe-drill` on a dispatch; `probe-edge` pins `cotel-health-probe-drill-edge`. The default is the `/healthz` subject's marker, shared by the scheduled job and `~/ops/cotel-healthz.sh` on the Pi | | `PC_ALERT_MAX_AGE_H` | `6` | Hours an open alert stays a dedup target on a red tick. `0` always dedups. Both scheduled jobs set `0` | | `PC_STILL_RED_WAKE` | `1` | Whether a red tick wakes the assignee of an already-open alert. `0` silences that wake and exits 0. Both scheduled jobs set `0` | | `PC_RUN_ID` | unset | Sent as `X-Paperclip-Run-Id` when set | From 0262a28fca7bb964954694e2d73ceb8b02deae38 Mon Sep 17 00:00:00 2001 From: Wayland Date: Mon, 19 Oct 2026 14:39:01 +0200 Subject: [PATCH 5/6] fix(health): assert snapshots on the deploy host, and catch the stall it saw MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two corrections from review, both about the scheduled watcher being weaker than the thing it was granted to replace. `SNAPSHOT_CHECK=require` on the job. It was running on the default `auto`, which pages only on an affirmative snapshot failure and stays green when the instance makes no claim at all. That default is written for a local checkout, where snapshots ship disabled; on the deploy host the same silence is a rolled-back binary that predates the contract, a worker that never started, or an `/api/v1/health` that has stopped answering. Under `auto` all three read as health, which is precisely the blindness this watcher exists to end — and the documented by-hand recipe for this host already passes `require`. A scheduled watcher must not be weaker than that. Verified green against production with the job's own settings: ingest age 3s, snapshot 3529s old, exit 0. The scheduled ingest staleness window goes from 24h to 10h. The evidence under the old number argued against it: ordinary overnight silence here measures six and seven hours, the only real stall ever recorded measured twelve, and a 24h window clears both — including the stall, which is the single measured instance of the class the threshold exists to catch. Ten hours sits between the two measured ends. The cost is now stated where the number is: a genuinely idle day can cross ten hours and page, bounded by the dedup pair to one ticket, and that is the better failure of the two. ADR-0026 records both decisions; the threshold section no longer concludes the opposite of what its own histogram shows. Co-Authored-By: Wayland Co-Authored-By: Claude Opus 4.8 --- .github/workflows/health-probe.yml | 25 ++++++---- ...6-scheduled-loopback-probe-with-a-pager.md | 47 +++++++++++++++---- docs/operations/duckdb-snapshots.md | 12 +++-- docs/operations/health-probe.md | 30 +++++++----- 4 files changed, 81 insertions(+), 33 deletions(-) diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index ad0fbad..273d7eb 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -205,14 +205,23 @@ jobs: cancel-in-progress: true env: HEALTHZ_URL: ${{ inputs.loopback_url || 'http://127.0.0.1:8080/healthz' }} - # A day on the schedule, six hours by hand. Measured over a week on this - # instance, ordinary ingest silence reaches six and seven hours overnight, - # so a 6h window would page a quiet night several times a week; the - # longest silence ever measured here, 12h, was a real outage. A dispatch - # keeps the tighter window because a human is reading that answer. The - # snapshot window is unaffected — that worker ticks on its own timer and - # does not care how much traffic arrives. - STALE_AFTER_SECONDS: ${{ github.event_name == 'schedule' && '86400' || '21600' }} + # Ten hours on the schedule, six by hand. The window has to sit above the + # quiet nights and below the outages, and both ends are measured on this + # instance: ordinary overnight silence reaches six and seven hours, while + # the only real stall ever recorded here ran twelve. A day would clear + # every quiet night and also clear that stall, which is the one thing this + # threshold exists to catch. A dispatch keeps the tighter window because a + # human is reading that answer. The snapshot window is unaffected — that + # worker ticks on its own timer and does not care how much traffic + # arrives. + STALE_AFTER_SECONDS: ${{ github.event_name == 'schedule' && '36000' || '21600' }} + # The default, `auto`, pages only on an affirmative snapshot failure and + # stays green when the instance makes no claim at all — right for a local + # checkout where snapshots ship disabled, wrong here. On the deploy host + # an unreadable `/api/v1/health`, a missing `snapshot` field or a worker + # that never started are failures, and this is the watcher that exists to + # report them. `require` is that assertion. + SNAPSHOT_CHECK: require CONNECT_TIMEOUT: "15" PC_API_URL: ${{ secrets.PAPERCLIP_API_URL }} PC_API_TOKEN: ${{ secrets.PAPERCLIP_API_TOKEN }} diff --git a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md index c51c902..15db95a 100644 --- a/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md +++ b/docs/decisions/0026-scheduled-loopback-probe-with-a-pager.md @@ -42,7 +42,7 @@ watcher was retired. That made it the CEO's call, and the CEO granted it. pages — with its own dedup marker, the dedup pair that bounds one outage to one ticket, an in-run red confirmation, and a measured staleness threshold.** -Six things had to be true together; the first five copy `probe-ingest`, whose +Seven things had to be true together; the first five copy `probe-ingest`, whose shape the board accepted in the same form. 1. **Own marker: `[cotel-health-probe]`**, this half's historical one and the @@ -85,15 +85,29 @@ shape the board accepted in the same form. landing in that window would otherwise mint a ticket for a routine deploy. The streak cannot live between runs — dropped ticks mean "two runs in a row" is not a quantity this job can count. -6. **The ingest staleness threshold is 24 hours on the schedule, 6 by hand.** +6. **The ingest staleness threshold is 10 hours on the schedule, 6 by hand.** Measured, not assumed; see below. +7. **`SNAPSHOT_CHECK=require`**, not the default `auto`. The probe sorts the + snapshot claim into three classes, and the two strict ones are not in + question: `ok` is green and `red` — the worker reported a failure, or its + newest restore point is older than its own 12h window — pages under either + setting. What the setting decides is the third, `unverified`: the instance + makes no claim at all, because snapshots are disabled, because the worker has + never run, or because `/api/v1/health` could not be read. `auto` stays green + there, which is right for a local checkout, where snapshots ship disabled and + a pager would cry wolf on every developer. On the deploy host the same + silence means a rolled-back binary that predates the contract, a worker that + failed to start, or an endpoint that has stopped answering — and `auto` would + report all three as health. This watcher was granted to stop exactly that, so + it asserts what is true of this host: snapshots are expected here. A dispatch keeps everything it had: the drill marker, the run id as the call id, and the 6h threshold. Only the scheduled event gets the production marker and the longer window, by the same `github.event_name == 'schedule'` conditional -`probe-ingest` already uses. +`probe-ingest` already uses. `SNAPSHOT_CHECK` is not latched to the event — a +dispatch on this host is asking the same question about the same instance. -### Why 24 hours, and where the number came from +### Why 10 hours, and where the number came from `scripts/probe-healthz.sh` exits **3** when `newest_span_age_seconds` exceeds `STALE_AFTER_SECONDS`, and the job shipped with 6 hours — a threshold written @@ -108,12 +122,25 @@ gap 6h 2026-10-15 22:00 -> 04:00 gap 6h 2026-10-15 10:00 -> 16:00 ``` -Ordinary quiet nights on this box reach six and seven hours of ingest silence. -With a 6h window a scheduled pager would mint "stale ingest" tickets several -times a week on nothing at all — precisely the noise the board retired the -previous watcher for. A full day with no span is an assertion: the longest -silence ever measured here, 12 hours, was a genuine outage. So exit 3 stays a -paging class, with a threshold that has evidence under it. +The threshold has to clear the quiet nights and still catch the stalls, and +those two numbers are seven hours apart: ordinary overnight silence on this box +reaches six and seven hours, and the only genuine stall ever recorded here ran +twelve. A 6h window would mint "stale ingest" tickets several times a week on +nothing at all — precisely the noise the previous watcher was retired for. A +24h window was the first answer here and it was the wrong one, for the opposite +reason: it clears every quiet night, but it also clears the 2026-10-14 outage, +the single measured instance of the failure class this threshold exists to +catch. A backstop that cannot catch the one red in its own evidence is not +detection. + +**10 hours** sits between the two measured ends: 2026-10-14 would have paged +after ten hours, and no quiet night on record reaches it. The residual risk is +named rather than hidden — a genuinely idle day, a holiday or a weekend with no +agent running, can cross ten hours with nothing wrong, and that will page. The +dedup pair bounds the cost of being wrong to **one** ticket, closed by hand, and +a false page on a quiet day is a better failure than silence through a real +stall. If idle days turn out to page more than rarely, raise the number with the +histogram that showed it rather than by feel. This does not touch `SNAPSHOT_STALE_AFTER_SECONDS` (12h). The snapshot worker ticks on its own timer and does not care how much traffic arrives, so its window diff --git a/docs/operations/duckdb-snapshots.md b/docs/operations/duckdb-snapshots.md index 3ce3fee..ca1250c 100644 --- a/docs/operations/duckdb-snapshots.md +++ b/docs/operations/duckdb-snapshots.md @@ -84,8 +84,10 @@ minutes, best-effort, since 2026-10-19.** `scripts/probe-healthz.sh` reads the `snapshot` object — a red verdict pages under the `[cotel-health-probe]` marker on a schedule, as an issue whose assignment wakes an agent. It runs on the self-hosted runner on the deploy host, which is the only vantage point from -which `/api/v1/health` is reachable at all. The same probe still works by hand -from any machine on the LAN. See +which `/api/v1/health` is reachable at all, and it runs with +**`SNAPSHOT_CHECK=require`**: on that host the two ambiguous rows in the table +below are failures rather than a local default, so they page too. The same probe +still works by hand from any machine on the LAN. See [Production health probes](./health-probe#the-vantage-points) and [ADR-0026](../decisions/0026-scheduled-loopback-probe-with-a-pager.md). @@ -119,7 +121,11 @@ the alert path. The last two rows flip to red under `SNAPSHOT_CHECK=require`, which is the caller asserting "snapshots are expected on this instance". Default is `auto`: page only on affirmative evidence that the backup is broken. `SNAPSHOT_CHECK=off` -stops the probe asking at all. +stops the probe asking at all. The scheduled watcher sets `require`, because on +the deploy host "the instance makes no claim" is itself the finding — a +rolled-back binary that predates the contract, a worker that never started, or +an endpoint that has stopped answering all land in those two rows, and under +`auto` all three would read as health. ### The threshold diff --git a/docs/operations/health-probe.md b/docs/operations/health-probe.md index 00478b4..498abd7 100644 --- a/docs/operations/health-probe.md +++ b/docs/operations/health-probe.md @@ -91,7 +91,8 @@ as a drill. | Catches | DNS, a blanked tunnel token, the ingest route not served, Access appearing in front of ingest | process dead, crash loop, 503 DuckDB, stale/empty ingest, a snapshot worker that has stopped producing restore points | the dashboard host through Cloudflare | | Blind to | the database and ingest freshness: this check stops at the auth boundary; and its own dropped ticks | **its own host being down** — with the runner off the job queues, producing no colour at all; and its own dropped ticks | nothing in the path, but see the Access caveat below | | Access service token | **not needed** | not needed | required | -| Ingest staleness window | n/a — never reads `/healthz` | **24 h** on a schedule, 6 h on a dispatch | 6 h | +| Ingest staleness window | n/a — never reads `/healthz` | **10 h** on a schedule, 6 h on a dispatch | 6 h | +| Snapshot claim | n/a — never reads `/healthz` | **`SNAPSHOT_CHECK=require`**: "no claim" is red, because snapshots are expected on this host | `auto` — and `require` must never be set here, see the Access caveat | | Pages | **production** `[cotel-ingest-edge]` on a schedule, `[cotel-ingest-edge-drill]` on a dispatch | **production** `[cotel-health-probe]` on a schedule, `[cotel-health-probe-drill]` on a dispatch | `[cotel-health-probe-drill-edge]` — drill only | **Neither scheduled job may be moved to the other's runner.** `probe-ingest` @@ -153,12 +154,12 @@ it to **24**, and that number is measured rather than chosen: see A present JSON `null` is an empty database (never ingested), which is not the same as `0` (ingested just now) and not the same as a missing key. -### The staleness threshold is 24 hours on the schedule +### The staleness threshold is 10 hours on the schedule -Exit 3 — "ingest stale" — is a paging class, so its threshold decides how often -a quiet night mints a ticket. An hourly span histogram over one week of -production (`/api/v1/history?granularity=hour&range=week`) says 6 hours is not -an assertion about this instance: +Exit 3 — "ingest stale" — is a paging class, so its threshold decides both how +often a quiet night mints a ticket and whether a real stall is caught at all. An +hourly span histogram over one week of production +(`/api/v1/history?granularity=hour&range=week`) puts both ends on the record: ``` gap 12h 2026-10-14 09:00 -> 21:00 (the tunnel outage — a real red) @@ -169,13 +170,18 @@ gap 6h 2026-10-15 10:00 -> 16:00 Ordinary silence here reaches six and seven hours overnight, so a scheduled pager on the 6h window would have paged several times that week on nothing. A -full day with no span is an assertion — the longest silence ever measured on -this box, 12 hours, was the real tunnel outage. So the scheduled job sets -`STALE_AFTER_SECONDS=86400` and a dispatch keeps `21600`, where a human is +full day, the first number tried here, fails the other way: it clears every +quiet night but also clears the 2026-10-14 outage, which is the only measured +instance of the thing this threshold is for. So the scheduled job sets +`STALE_AFTER_SECONDS=36000` — ten hours, above every quiet night on record and +below the one real stall — and a dispatch keeps `21600`, where a human is reading the answer and a tighter question is more useful. -Moving this number is a decision about alert noise, and it needs the same -histogram taken again — not a guess. `SNAPSHOT_STALE_AFTER_SECONDS` is +What that costs is named rather than hidden: a genuinely idle day — a holiday, +a weekend with nothing running — can cross ten hours with nothing wrong, and it +will page. One ticket, not a stream, because the dedup pair bounds it; close it +by hand. If that starts happening more than rarely, raise the number with a +fresh histogram rather than by feel. `SNAPSHOT_STALE_AFTER_SECONDS` is unaffected either way: that worker ticks on its own timer and does not care how much traffic arrives. @@ -230,7 +236,7 @@ a probe-configuration problem), and the ingest probe, which never reads | Variable | Default | Role | |---|---|---| -| `SNAPSHOT_CHECK` | `auto` | `auto` pages on affirmative failure only; `require` also pages when the instance makes no snapshot claim; `off` never asks | +| `SNAPSHOT_CHECK` | `auto` | `auto` pages on affirmative failure only; `require` also pages when the instance makes no snapshot claim; `off` never asks. `probe-loopback` sets **`require`** — it only ever asks the deploy host, where snapshots are expected, so "no claim" is a failure there and not a local default | | `SNAPSHOT_STALE_AFTER_SECONDS` | `43200` | How old the newest snapshot may be. Two `COTEL_SNAPSHOT_INTERVAL` periods at the shipped `6h`; move it with the interval | | `API_HEALTH_URL` | derived from the `/healthz` URL | Override when the snapshot report is not at the same entry point | From d85aaa87acf8f85524d75d267ef68026859ae36d Mon Sep 17 00:00:00 2001 From: Daedalus Date: Tue, 20 Oct 2026 12:51:23 +0200 Subject: [PATCH 6/6] docs(health): the scheduled probe, not a hand run, now watches the snapshot ADR-0025's restore-point bullet said running probe-healthz.sh by hand was the only thing that could notice a snapshot stopping being refreshed. It was true when written and is not now: the bullet above it records the granted pager, and the scheduled probe-loopback run carries SNAPSHOT_CHECK=require. Also rewraps one comment line in the workflow header that the measured-rate edit left long. Co-Authored-By: Daedalus Co-Authored-By: Claude Opus 5 --- .github/workflows/health-probe.yml | 6 +++--- .../0025-best-effort-ingest-watcher-in-actions.md | 7 +++++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/.github/workflows/health-probe.yml b/.github/workflows/health-probe.yml index 888e1f4..43ce62d 100644 --- a/.github/workflows/health-probe.yml +++ b/.github/workflows/health-probe.yml @@ -4,9 +4,9 @@ name: Health probe # monitoring**. GitHub schedules public repositories on a best-effort basis and # *drops* ticks rather than delaying them — measured in this repository over a # full day, 4 runs out of the 91 expected (4.4%), about one look every 5.7 -# hours — so the detection delay either one buys is unbounded. They are strictly better than nothing, and that is the -# whole of what they promise. See docs/operations/health-probe.md, ADR-0025 and -# ADR-0026. +# hours — so the detection delay either one buys is unbounded. They are +# strictly better than nothing, and that is the whole of what they promise. See +# docs/operations/health-probe.md, ADR-0025 and ADR-0026. # # `probe-ingest` asks the public OTLP ingest URL from outside every network # cotel lives in. It is `ubuntu-latest` and must never be self-hosted: an diff --git a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md index cdc22bf..d9735f0 100644 --- a/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md +++ b/docs/decisions/0025-best-effort-ingest-watcher-in-actions.md @@ -192,8 +192,11 @@ bound, and the first person to rely on that is the one holding a silent outage. the restore point rotted is gone along with the second copy. What remains is one host writing snapshots to its own disk, which makes a pager on `/api/v1/health` the only thing that could notice the restore point stopping - being refreshed, and makes running `scripts/probe-healthz.sh` by hand the only - thing that does. + being refreshed. Running `scripts/probe-healthz.sh` by hand was the only + thing that did it until the grant above: the scheduled `probe-loopback` run + asks the same endpoint with `SNAPSHOT_CHECK=require`, so a stalled snapshot + now pages — as often as GitHub delivers a tick, which is the measured 4.4% of + them. - **Reaching `/healthz` through Cloudflare Access is closed, and not on technical grounds.** The `CF_ACCESS_CLIENT_ID` / `CF_ACCESS_CLIENT_SECRET` pair still gets HTTP 302 from `https://cotel.aignite.pl/healthz` (runs