[erikdarlingdata]

Claude posting for Erik Darling.

Closes #6927.

Draft on purpose: it must not merge before the current release is published.

Why

The blocking colour on the Fleet Overview, the Overview tab and the Daily Health Calendar came from compile-time constants (warn at 5 reports per hour or a 10 s block, critical at 20 per hour or a 60 s block). An operator who tunes the blocking alert to their workload still saw red. The fleet totals row also coloured the Deadlocks tile critical and the Blocking tile warning on any non-zero count.

What changes

• PerformanceMonitor.Common: new BlockingBandThresholds (four tiers, clamped on read, defaults equal today's constants) and floors and ceilings beside the deadlock ones. ServerHealthClassifier.BlockingSeverity takes an optional tiers argument, ServerHealthMetrics.BlockingBandThresholds and DailyHealthThresholds.BlockingBands carry them. Lite passes nothing and keeps the constants.
• Store: migration V228 adds blocking_warn_per_hour (5), blocking_critical_per_hour (20), blocking_warn_wait_seconds (10) and blocking_critical_wait_seconds (60) to config_alert_settings, idempotent (ADD COLUMN IF NOT EXISTS). StorageVersion is 228, the viewer schema probe and MapProbedSchemaVersion have a V228 arm, and the rung rewind list has the V228 entry. No provision-roles.sql change: V120 added none either, the existing table-level grants cover new columns.
• Service: DarlingConfig, DarlingAlertSettings.BlockingBandThresholds, the store seed and read in StoreConfigProvider, get_alert_settings (health_bands.blocking_*), update_alert_settings (bounds 1 to 10000 per hour and 1 to 86400 seconds, critical below warn refused in the same body or against the stored other value, nothing written on refusal).
• Readers now using the store values through one shared read (BlockingBandStore): fleet cards and get_fleet_overview (including the worst-first re-band), the fleet sweep and its evidence thresholds, get_daily_health and the calendar, the Viewer Overview card, calendar and Settings window (four boxes, validated, restore defaults).
• Web: Admin page lists the four fields (read only, as the whole Alert Settings tab is). The fleet roll-up carries total_blocking_severity and total_deadlock_severity, and each is the worst band among the cards for that signal, the classification the cards already show, computed where the roll-up is built. fleet.js colours the two tiles from them. A fleet sum is never banded as one server's rate: ten servers at 3 events an hour (no card even Warning at the default 5) no longer turn the tile Critical.

Test plan

Runs on a throwaway PostgreSQL cluster (port 55927, stopped afterwards), DARLING_CONFIG pointing at a scratch file.

• BlockingBandTiersTests, ServerHealthClassifierTests, DailyHealthBandTests, DarlingMcpAlertToolsTests, ConfigSeedStatementArityTests, plus the new BlockingBandSettingsLiveTests, ClientTimeoutsSettingsSurfaceTests and ClientTimeoutsSettingsLiveTests: Total 165, Failed 0.
• 23 neighbouring classes (fleet reader, fleet DTO JSON, fleet sweep, deadlock band, admin page, MCP alert and fleet tool surfaces, schema version gate, census and ratchet classes): Total 367, Failed 0.
• Stage=Guard trait: Total 4478, Failed 3. The three are the dense-ladder checks (CollectorStallProbeStoreTests, StoreLogSelfMonitoringStoreTests, PgCpuCapacityHeadroomTests TheRungIsRegistered...). They fail only because this branch has V228 and dev tops at V226: V227 is Hold alert threshold overrides for a server tag (#6926) #6939's. That was before the merge with V227 below, and CI runs them again on the merged head.
• Fleet totals test: ten servers at 3 events an hour give a tile that is not Critical, and Critical when one card is Critical. New BlockingBandSettingsLiveTests covers the V228 upgrade on a store whose settings row exists (defaults 5, 20, 10, 60, the operator's deadlock tier untouched) and the update_alert_settings / get_alert_settings round trip, including critical-below-warn refused with nothing written (in one body, and lone against the stored value) and a value below 1 refused.
• Existing pins moved from 86 to 90 settings columns: ClientTimeoutsSettingsSurfaceTests only.
• Browser (playwright-cli, the service built from this branch against the scratch cluster, two screenshots, one wide and one at phone width): with ten servers at 3 events an hour the Blocking tile at 30 is not coloured (defaults), and Warning with the bands set to 2 and 6 (every card Warning), and Critical once one card passes 6. At 390 px wide the tiles wrap and nothing scrolls sideways. The Admin page shows the four fields at 5, 20, 10 and 60, then 2, 6, 30 and 120 after a save through update_alert_settings and a reload. The Admin page cannot set them (it is read only), so the save and the refusals went through the MCP tool: critical 1 with warn 2 and warn 200 with critical 100 both came back invalid with nothing written.
• Merged with V227 (Hold alert threshold overrides for a server tag (#6926) #6939, now on dev), then merged dev again. StorageVersion.SchemaVersion is 228, V228 sits after V227, both rungs keep their rewind artifacts, probe sentinels and census entries. V228 adds only constant-default NOT NULL columns and is not a data-moving rung, so the census needed no new declaration. On a throwaway cluster: MigrationDataMovingRungCensusPins, McpReadCommandTimeoutTests, RungRewindArtifacts, ViewerSchemaVersionGateTests, every *Census*, *Ratchet*, *Migration*, *Rung* and *SchemaVersion* class, BlockingBand*, ClientTimeoutsSettingsSurfaceTests, DarlingMcpAlertToolsTests, ThresholdOverride*, ViewerSweepPressureLiveTests: Total 1385, Failed 0. The three dense-ladder failures above no longer apply.
• One test fix from the merge: ThresholdOverrideTagScopeTests.V227_OnAStoreThatHoldsServerOverrides_KeepsThem_AndTakesTagRows asserted the newest stamped version was 227 and now compares with StorageVersion.SchemaVersion.
• Full suite: not run on this laptop, CI runs it.
• Viewer Settings window (four boxes, validation, restore defaults) was not driven by hand, no UI automation for it here.

Considered, not added

• Per-server and per-tag values: out of scope as decided.
• A web form to set the four values: the Alert Settings tab is read only for every field, so the four follow it.
• A deadlock-pair critical-below-warn refusal: the deadlock pair has accepted it since V120 and this change does not alter that.
• Combining the fleet blocking tile's number across servers into one rate: shown above to disagree with the cards, so the tile takes the worst card instead.

Notes for whoever merges

• Migration numbering: V227 is Hold alert threshold overrides for a server tag (#6926) #6939, V229 is [FEATURE] Notification routes: match on alert severity, so one tag can send Critical and Warning to different channels #6928. This PR keeps V228 and must land after Hold alert threshold overrides for a server tag (#6926) #6939 so the ladder is dense.
• Decision to check: critical below warn is refused for the blocking pairs (the deadlock pair accepts it), because warn below critical was asked for. Equal values are accepted, like the Runnable Queue and Client Timeouts pairs.
• The only build warning is CA1068 in ViewerDataService.CollectionHealth.cs, a file this change does not touch.

Click-through attempt (Viewer Settings window, Overview card, calendar)

Not done. The viewer cannot run as a second copy: it is single-instance, and a launch with --config pointing at a throwaway store handed off to the viewer already running on this machine and exited (its log line: Handoff ... -> Surface). That running viewer was not touched beyond that surface request, and no connection or setting in the real profile was changed. So these remain to be clicked, on a machine with no other viewer running:

• Settings window: the four blocking boxes load the stored values, save, and reload after reopening.
• Critical below warn is refused with a clear message in the window.
• The Overview card and the calendar band follow the saved tiers.

What was checked instead: the service from this branch on a throwaway PostgreSQL cluster (own config file, first log line named it). A fresh store came up with the V228 columns at 5, 20, 10 and 60. update_alert_settings with health_bands.blocking_warn_per_hour 8 and blocking_critical_per_hour 4 was refused with "must not be lower than", nothing written. A valid pair (8/30 per hour, 15/90 seconds) was accepted and read back from the store.

Changes after a code check

• Removed DarlingAlertSettings.BlockingBandThresholds. It had no reader: the service, the viewer and the MCP tools all read the tiers through BlockingBandStore. The config fields it wrapped stay, because the seed insert and the store load still bind them.
• The fleet Deadlocks totals tile used to turn Critical on any non-zero total. It now follows the worst card band, the same as the Blocking tile and the cards.
• Ran BlockingBandTiersTests and ServerHealthClassifierTests: Total 75, Failed 0. Nothing else in the tests or the source names the removed property. Not re-run after the final rebase onto the branch.