[erikdarlingdata]

Claude posting for Erik Darling.

Closes #6928.

Why

A notification route could not be limited by severity, so production warnings could not be kept out of an on-call channel.

What changes

Store (V229). config.config_notification_routes.severities is both, critical or warning, default both, so every existing route keeps its effect. V229 also adds config.external_alert_state.fired_severities and config.investigation_request_alerts.severity (see Resolutions and the agent gate below). The rung is idempotent and has its rewind artifact. StorageVersion.SchemaVersion is 229.

Matching. NotificationRouter.Resolve and ResolveAgent take an optional severity. A route whose filter excludes the alert's tier is dropped first, then the existing order and tie-breaks run unchanged, ending at the default channels. A caller with no severity is matched by every route, as before, so Lite is unaffected. The route cache key includes the severity filter, so an edit of only that filter reloads the cached routes and takes effect without a restart.

Which tier an alert has. The tier graded at the fire site when there is one, else the metric's own badge. Red is Critical. Every other badge counts as Warning, so a Warning-only route also takes alerts that are not Critical.

Resolutions and escalation.

• A resolution goes to the channels the fire went to. It does not re-match routes, so a filter change never moves a clear.
• When an open Warning alert climbs to Critical, it is delivered again as a Critical fire. The clear is then sent to every destination the episode fired to (PagerDuty per routing key, plus Teams, Slack, webhook and email), so a Warning incident is not left open on the first route's channel.
• The external-alert PagerDuty close used to re-match the route by metric alone. It now closes at every tier the episode fired at, read from fired_severities.

Agent-action gate. Each alert row on an investigation request now records its tier, and the gate resolves each alert's route at that tier. Before, a Critical alert dispatched under a fleet route at Act could be lowered to Off by a server route that is Warning only. A row written before V229 resolves as before. The gate still only lowers a request's level.

Journal. A change to severities on a route with an agent setting is journaled and shown, like a match or scope change, and a non-operator is refused the same way.

MCP. create_notification_route and update_notification_route take severities (any case, anything else is invalid). get_notification_routes shows it, and the undo answers carry it.

Viewer. The route editor has a Severity choice (Both, Critical only, Warning only) and the routes window has a Severity column. The column is read only on a store at V229 or newer. Below that every route shows as Both.

Web. The Routes list has a Severity column and the route editor a Severity choice. The service accepts severities on create and edit, keeps the saved value when an edit leaves it out, and refuses any other word with a clear message.

Web route edit fix (also in v3.11.0). Saving an edit to an existing route from the web page always failed with HTTP 400 "expected_modified_at is required". The edit form dropped the route's last-saved time when it read its fields back. One line in admin-agent-setup.js fixes it. This is older than this PR (since #6274), and is fixed here because it is the same code.

Docs. alerts-and-notifications.md, configuration-mcp.md, and the V229 rollback line in docs/darling/admin.md.

Route severity is Darling only. Lite's routes have no severities field, and the Lite and Darling route field check lists it as Darling only.

Tests

• New: NotificationRouteSeverityTests, NotificationRouteSeverityLiveTests, AgentActionGateSeverityLiveTests, ViewerNotificationRouteSeveritiesTests, a resolution test in AlertResolutionDeliveryTests, a drain test in ExternalAlertDrainLiveTests, and AdminPendingNotesTests.SavingAnEditedRoute_SendsTheVersionTheFormRead_AndTheChosenSeverity for the web edit fix (it fails without the fix).
• The agent-gate test fails when the fix is taken out (the request is refused as not Level 2) and passes with it.
• After merging V227 and V228 under V229: the census, ladder, rung, migration, schema-version, MCP read-timeout, route severity, pending notes, agent setup and agent-gate classes, Total 1466, Failed 0, on a throwaway PostgreSQL cluster.
• Browser, against the service built from this branch on a throwaway store: the Routes list and editor at 1400 px and 390 px. Create Critical only, edit to Warning only, then to Both, and the list shows each. A bad word is refused with "severities is critical, warning or both." No sideways page scroll at phone width.
• After merging dev (V227 and V228 as squashed there, V229 on top, dev's text kept for both): the rung census, ladder, rewind artifact, schema-version gate, route severity, threshold override and blocking band classes, Total 1107, Failed 0, on a throwaway PostgreSQL cluster. The PR reads MERGEABLE.
• After the first CI run: DarlingRouteCacheReloadTests, InvestigationFeedbackMcpLiveTests and the other Investigation*Live* classes, ManagedRoles, ProvisionRoles, NotificationRouteSeverity*, AgentActionGateSeverityLiveTests and AdminPageTests on a throwaway store, Failed 0. The route cache key gained the severity filter, and the mcp role was granted INSERT on investigation_request_alerts.severity, which the practice copy writes. The routes table test and the managed-roles grant pin name the new column.
• CI runs the full suite.

Merge order

V227 (#6939) and V228 (#6938) are on dev. This PR is V229 and goes after them.

Considered, not added

• Hiding the viewer's Severity choice on a store below V229: the viewer's connect gate already refuses a store below the required version.
• Rebuilding provisioned roles on a store that kept the old route functions: the managed batch drops them, and a bring-your-own store gets the same step from the existing provisioning note.