Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion docs/decisions/0025-best-effort-ingest-watcher-in-actions.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,23 @@ bound, and the first person to rely on that is the one holding a silent outage.

- **The detection delay is unbounded**, and no amount of tuning the cron
changes that. 15 minutes is a ceiling on how often it *could* look.
- **Measured on this watcher, over its first day: 4 ticks against 91 expected,
4.4%.** The window was 2026-10-19 05:30Z to 2026-10-20 04:06Z, the ticks
landed at 11:07, 17:28, 21:42 and 00:42Z, and the gaps between them were
6.4 h, 4.2 h and 3.0 h. Registration lag does not explain it: counting only
from the first tick gives 3 of 67, the same 4.5%. So the real cadence of this
watcher is **roughly one look every 5.7 hours**, not every 15 minutes, and
the worst gap observed so far is 6.4 hours.

This is worse as a ratio than the `cron: "17 * * * *"` sample quoted above
(1 of 9, 11%) and slightly better in absolute terms (one look per 5.7 h
against one per 9 h), which is the only comparison that matters for
detection. **It does not reopen the decision.** The acceptance bar this
watcher was built against is catching a silent outage *the same day*, after
six days of blindness was the original failure; a worst case of 6.4 hours
clears that bar. What it does invalidate is the number a reader infers from
the cron expression, so quote the measured cadence and not `*/15` wherever
this watcher's latency is described.
- **Whether it ran at all is a thing to measure, repeatedly.** The one-line
check above belongs in any review of this watcher; empty output is the
answer, not an error.
Expand Down Expand Up @@ -121,7 +138,12 @@ bound, and the first person to rely on that is the one holding a silent outage.
is three edits, not one — its pager steps hang on `inputs.page`, which
`schedule` does not supply, and it does not set the `PC_ALERT_MAX_AGE_H=0` /
`PC_STILL_RED_WAKE=0` pair above, so a pager wired up naively would mint a
ticket per tick, 96 a day for one outage. Scheduling it *without* a pager was
ticket per tick. That was first written here as 96 a day for one outage,
which is the nominal figure and about 24 times the real one: at the delivery
rate measured above it is nearer 4 a day. The pair is still required, because
the requirement is one ticket per outage and not a tolerable number of them,
but the case for it rests on the rule rather than on the scare figure.
Scheduling it *without* a pager was
considered and rejected: a red run nobody is subscribed to is the appearance
of detection, not detection, and this record already warns against exactly
that trade one bullet up. So the choice is binary — page from inside the LAN,
Expand Down
Loading