fix: Stop paging Slack for customer-owned DNSRecordSet states - #157
Merged
Merged
Conversation
DNSRecordSetNotAccepted fires on a condition its audience mostly cannot act on. Of the 62 records currently not-Accepted in staging, 58 name a DNSZone withheld pending domain-ownership verification and 3 name one claimed by another resource. Both are the system behaving correctly and both persist until the project owner acts, so the alert is a standing description of customer state rather than a call to an operator. It carries severity: warning and service: dns, which the Alertmanager route sends to slack-warning, so all 62 post to Slack and stay there. Drop it to severity: info. The route tree blackholes severity=info when no slack_channel label is set, which is how the other 19 info alerts on the platform stay quiet, and the alert keeps firing in vmalert and rendering in Grafana for anyone looking at a specific project. Confirmed with amtool against the live config: severity=warning resolves to slack-warning, severity=info to blackhole. Nothing here was ever paging. grafana-oncall is reachable only at severity=critical or by two alertnames by exact match, and these alerts are neither. The two drift alerts stay at severity: warning and keep their Slack route, since an orphaned or unreplicated record is an operator failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
scotwells
marked this pull request as ready for review
September 12, 2026 17:18
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DNSRecordSetNotAcceptedposts to Slack for a state only the customer can clear. Of the 62 records currently not-Accepted in staging, 58 are waiting on domain-ownership verification and 3 on a zone claimed by another resource — the system working as designed, not an operator failure.Dropping it to
severity: infokeeps the alert firing and visible in vmalert and Grafana, but the Alertmanager route blackholes it instead of posting. That's the same convention the other 19 info-severity alerts on the platform use.Verified with
amtoolagainst the live staging config:severity=warning(today)slack-warningseverity=info(this PR)blackholeNothing here was ever paging on-call —
grafana-oncallis reachable only atseverity=criticalor by two alertnames matched exactly. The two drift alerts keepseverity: warningand their Slack route, since an orphaned or unreplicated record is an operator failure.One gap this leaves: exactly one of the 62 is a genuine bug — a record naming a DNSZone that doesn't exist — and it goes quiet along with the rest. Worth a separate rule that fires only on that case.