Add native AWS CloudWatch and Azure Monitor alert ingress, plus alert write-back mutations

[sarora-eightfold]

Adds two native alert-ingress integrations so CloudWatch and Azure Monitor can reach GoAlert directly, with no Lambda, forwarder or middleware. They are deliberate siblings: same handler shape, same payload-mapper structure, same dedup discipline, same smoke-test layout.

Neither can use the existing generic endpoint, for different reasons — SNS is blocked on transport and handshake, Azure on payload shape.

Also adds two GraphQL mutations that let an external system write back onto an existing alert, built to support a Jira Automation rule that opens a ticket per alert, assigns it to the current on-call, and needs to avoid opening a second ticket for the same alert.

ticket: https://eightfoldai.atlassian.net/browse/ENG-206319

AWS CloudWatch (via SNS)

CloudWatch alarms reach us through an SNS topic, but GoAlert could not be subscribed to one directly for three independent reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription and nothing did that, so zero messages were ever delivered; the SNS envelope has no summary field and carries the alarm as a JSON string inside Message; and SNS always sends text/plain, which genericapi.ServeCreateAlert ignores because it only unmarshals application/json. That last point is upstream target#4463.

Add a cloudwatch integration key type and a handler at POST /api/v2/cloudwatch/incoming that performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly with no Lambda or forwarder in between.

The key type is a full integration rather than a reuse of TypeGeneric because the webhook URL shown in the UI is generated server-side from the key type alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a /api/v2/generic/incoming URL that silently fails to confirm as an SNS subscription -- reproducing the exact bug this change fixes. Because the UI is driven entirely by IntegrationKeyTypes and Href, no frontend changes are needed.

Notable implementation details:

• The body is parsed regardless of Content-Type, and bounded with MaxBytesReader rather than io.LimitReader so an oversized body yields a 413 instead of silently truncating into a misleading 400.
• Signature verification is split into pure functions (canonical string, verify, cert parse) so they are testable with no I/O. Subject is a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string cannot distinguish from an empty value.
• Both outbound fetches are host-allowlisted with an anchored pattern, and the client blocks redirects: the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses.
• The signing-cert cache is bounded with FIFO eviction because the cert URL path is attacker-supplied and would otherwise grow without limit.
• A freshness window on the signed Timestamp bounds replay of a captured envelope. Tradeoff: retries arriving over an hour late are rejected.
• Dedup is hex sha256(AlarmName), matching the CloudWatch alarm Lambdas that post to PagerDuty, so one alarm cannot produce two alerts across ingress paths. It is never nil, since a nil dedup silently falls back to a content hash that changes on every state transition and would break both idempotency and the OK close.
• NewStateReason is capped before assembling details so a verbose reason cannot push AlarmDescription, which carries the runbook URL, past the length limit.
• INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure so SNS retries only when a retry could help.

---

Azure Monitor

Azure Monitor delivers alerts by having an action group POST a webhook. Azure sends application/json so it clears the content-type gate that blocks SNS, but ServeCreateAlert expects a flat body while Azure nests everything under data.essentials / data.alertContext with no top-level summary — so every delivery would create an alert with a blank summary rather than an error. Nothing maps Azure's alertId onto the dedup field either.

Adds an azureMonitor key type and a handler at POST /api/v2/azuremonitor/incoming.

Unlike SNS there is no subscription handshake and no signature, so this handler makes no outbound requests at all — there is no analogue of cloudwatch's host allowlist or certificate cache. The integration key in the URL is therefore the only credential, which makes the webhook URL credential-grade; the docs say so explicitly for this key type.

Parsing:

• Only the common alert schema is accepted. A legacy-schema payload is rejected with a message naming the fix (enable the common alert schema on the receiver) rather than degraded to the fallback, which would produce content-free alerts with no indication why. 400 rather than 5xx, since Azure does not retry 4xx and a misconfigured receiver is a permanent condition.
• Dispatch is on the presence of condition.allOf, not a conditionType allowlist. Every metric and log criteria shape shares that envelope, so one code path covers SingleResourceMultipleMetricCriteria, MultipleResourceMultipleMetricCriteria (used by multi-resource and resource-group-scoped rules), DynamicThresholdCriteria and WebtestLocationAvailabilityCriteria. conditionType still selects the two behaviours that genuinely differ: suppressing the dynamic threshold, which is a sensitivity artifact rather than a limit, and the log query/link lines.
• Prometheus rule groups get their own branch — that shape carries no conditionType and no condition, only expression/labels/annotations.
• Service Health and activity-log payloads render from properties, with HTML stripped and string-containing-JSON fields left opaque.
• Anything unrecognised builds a best-effort alert from essentials, which is present on every payload regardless of type, and logs the signalType/monitorService/conditionType triple so a newly-routed alert type announces itself instead of silently producing thin alerts.

signalType is deliberately not the discriminator. Platform and Prometheus metric alerts both report signalType: "Metric", and Log Alerts V2, Azure Backup and ActivityLog Administrative all report "Log". Branching on it would feed unrelated payloads to the wrong renderer.

Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert object carries the whole lifecycle and the Fired and Resolved deliveries share an alertId — which is what lets the Resolved delivery close the alert its Fired delivery opened. originAlertId is deliberately not used: it is per-rule for metric alerts, so a single missed close would hold the dedup key and mute that rule permanently. Status comes from essentials.monitorCondition, never alertContext.status, which can disagree with it because the underlying incident resolved while the alert fired.

A json.UnmarshalTypeError on an individual field is tolerated rather than fatal. Azure documents threshold and dimension values as strings but is not consistent across shapes, and a hard failure means a 400 — which Azure does not retry — so one oddly-typed field would lose the page instead of one value.

---

Verification

CloudWatch — verified end-to-end against live AWS SNS: the subscription confirms, a real alarm creates one alert with the runbook URL preserved in details, and re-delivery is suppressed as a duplicate.

Azure Monitor — the target tenant's inventory was measured path-agnostically across all resource types, rather than by assuming where the action-group reference lives (it differs per rule type, which is how Prometheus was initially missed): 495 metric rules, 8 log (V2) rules and 3 Prometheus rule groups route to a PagerDuty action group. All three shapes are natively parsed.

Both packages have table-driven unit tests — the first unit tests in any GoAlert ingress package — plus a smoke test each. go build ./..., go vet and gofmt are clean and the full suite passes.

New CW option in integrations:

---

Alert write-back mutations (for external automation)

An external system that reacts to a GoAlert alert (e.g. a Jira Automation rule that opens a ticket for it) had no way to write anything back onto that alert. updateAlerts only changes status/noiseReason, and meta/details are otherwise set once, at createAlert time. Two small additions close that gap.

setAlertMetadata(input: SetAlertMetadataInput!): Boolean! — merges the given key/value pairs into an alert's existing metadata rather than replacing it. The underlying store call (SetMetadataTx) does a wholesale INSERT ... ON CONFLICT DO UPDATE SET metadata = $2, so the resolver reads existing metadata first and folds the new keys in, in one transaction — calling the store method directly with just the new key would silently erase everything an ingress integration had already written at creation time.

appendAlertDetails(input: AppendAlertDetailsInput!): Boolean! — appends text to an alert's existing Details, separated by a blank line. This is the one that's actually visible: alert metadata is written correctly by the mutation above but is never surfaced anywhere in the GoAlert web UI, only reachable via the API — appendAlertDetails writes to the same free-text field already shown on the alert page. No merge-by-key is possible here since Details is free text, so the resolver reads the current row (AlertStore.FindOne) and appends, truncating to MaxDetailsLength rather than erroring if the combined text overflows -- losing the append over a length limit is a worse outcome than truncating, since the alert is already real and actionable. Required one new store method (SetDetailsTx) and SQL query, since no existing mutation had ever touched Details post-creation.

Both share the same closed-alert guard as the existing metadata store method (rejected once the alert is closed, no partial effect) and the same permission model (permission.User or permission.Service, service callers scoped to their own alerts) — reachable via a scoped GraphQL API key with no user session, which is how the external automation authenticates.

Two unrelated fixes landed alongside this while building it out:

• A source link in alert details. CloudWatch alerts now render a Console: line deep-linking to the alarm in the AWS Console, built from its ARN. Azure alerts already rendered essentials.investigationLink when present; added a Portal: fallback (a generic ARM-resource-ID deep link) for when it's absent, which per Microsoft's docs requires limited-preview registration and so is the common case, not the edge case.
• A real cross-region bug the above surfaced. CloudWatch's Region: field and metadata were derived from the SNS topic's ARN, not the alarm's own ARN -- fine only by coincidence, since the deployed architecture is one topic receiving alarms from every region. An alarm firing in a different region than its topic would silently show the wrong region, while the new Console link (which always parses the alarm's own ARN) showed the correct one -- the same alert, two fields, disagreeing. Fixed to prefer the alarm's own ARN, with a dedicated cross-region test proving the two now agree.

Smoke-tested: merge-not-replace and overwrite-single-key for metadata; append-without-disturbing-existing-content, stacking on a second append, and truncation-not-rejection for details; closed-alert and nonexistent-alert rejection for both, with no partial effect.