[erikdarlingdata]

Claude posting for Erik Darling.

Fixes #5644 (W18 is fixed by #5704).

Summary

This PR fixes the web UI audit items W1-W22 of #5644 across the Darling service, Lite, the WPF viewer and the web pages. The table states the FINAL behaviour after the review and late fix lanes. It was deployed to DARLING01 and each W item was checked live (G6a-d); the final-review fixes (H lanes) came after those checks, and the Deploy and live checks section says what each check showed and what only tests cover.

Fixes by item

The fix column is the FINAL behaviour; lanes A-F are first-pass, G1a-G1d, G4a and G4b review lanes, H1, H2, H4a and H4b late fix lanes. Lite has no web UI.

Item	What was wrong	Fix (final behaviour)	Tests
W1	Every blocked-process report counted as one blocking event. SQL Server writes a report every blocked process threshold seconds (5 s lowest), so one 1-minute wait counted about 11 times in the alert value, the fleet "Blocking N" tile and the analysis BLOCKING_EVENTS fact.	One definition, PerformanceMonitor.Common/BlockingWaits.cs: one event = one blocked session, blocker and wait start (derived start = report time minus wait_time_ms). Reports of one pair whose derived starts are within 2 s of the previous one are one wait; the longest wait time is kept. Written twice: BlockingWaits.Collapse (C#) and BlockingWaits.WaitsCteSql (one SQL text for PostgreSQL and DuckDB). Counts that now count waits: Darling FleetBlockingSql; analysis PgFactCollector.BlockingSql / BlockingSqlSkippingSeparate (also the Azure-master scoped fleet read); get_server_summary (DarlingHealthReader.ServerSummaryBlockingSql); WPF viewer (FleetTotalsSql, ServerSummaryBlockingSql, FleetMasterUnscopedSql); Lite analysis (DuckDbFactCollector.Waits.cs) and overview (LocalDataService.Overview.cs, read by Lite's MCP summary); the DMV-snapshot fallback; Lite's server tab Blocking badge (LocalDataService.GetAlertCountsAsync, XE-then-DMV fallback and its windows kept); and the daily summary (Darling DailySummarySql.RangeSql, which serves the viewer calendar and MCP get_daily_health / get_daily_summary*, and Lite DailySummaryRangeSql), bucketed on the day of the wait's longest report with the blocking_events field unchanged (RangeSql is now static readonly; the Azure master scope rewrite still matches its bpr line). G1b: DuckDB uses epoch_us / 1000.0 (was epoch_ms) so all three agree on sub-millisecond starts; a report with no event or wait time is its own wait; window orders are NULLS LAST, the longest-wait pick wait_time_ms DESC NULLS LAST, event_time DESC NULLS LAST. Darling/README.md (blockingCountThreshold row, alert table row) and the BlockingCountThreshold remark now say blocking waits.	BlockingWaitCountTests, Lite.Tests/BlockingWaitDuckDbTests (7, plus 2 for the badge and the daily summary), DailySummaryBlockingWaitsLiveTests (live PostgreSQL: report arm, DMV arm, master scope), PerformanceCalendarDataTests and AzureMasterDailySummaryScopeTests (seeds adjusted).
W2	The alert value was the raw XE row count while the card's "Blocking chain (xN)" came from GroupBlocking over a differently built list, so the audit saw an off-by-one (43 vs "(x42)"). Inferred, not reproduced (see decisions).	The value and the card now come from one list: AlertContextBuilders.BlockingWaitRows (exclusions, collapse per source, XE-preferred), shared by Darling and Lite via PerformanceMonitor.Alerting. The alert value is its Count; BuildBlockingContext and BlockingIncidents group the same list, so chain counts add up to the value ("Wait Range" is over longest waits). G1d: hasXeReports is decided on the unfiltered rows, before the excluded-database filter, so XE reports only in an excluded database give an empty set and no fire (before, it fell back to DMV waits and fired); this matches Lite's overview and the fleet tile.	BlockingWaitCountTests (42 + 1 = 43; XE only in an excluded database), LiteAlertForwardingTests.
W3	CPU samples stored 82-87 times an hour for 60 records. The computed sample_time (SYSDATETIME minus ms_ticks - timestamp) drifts a few ms between reads, so the newest record lands above the stored watermark and is stored again (sql2022: 8 ms over 68 s).	Shared collector (Darling and Lite), no schema change: the ring-buffer dedup drops anything within 20 s above the watermark (RingBufferResampleTolerance, records are 60 s apart; both the local and UTC watermark frames). Lanes B, G1b.	CpuUtilizationCollectorDefinitionTests (8 ms and 3 s drift dropped, record 60 s later kept, both frames), CollectorTimestampFrameTests pin.
W4	FinOps "Top Databases by Avg CPU / Execution" dropped the busiest database: Convert.ToInt64 on avg_cpu_ms (the SQL yields DECIMAL(19,2)) tied 2.3, 1.95 and 1.56 as 2.	TopResourceConsumer and the viewer/Lite rows gain decimal AvgCpuMs; the by-average grid ranks on it (CpuTimeMs stays only as the whole-ms rounding). MCP top_by_avg.avg_cpu_ms carries two decimals (tool description updated); web FinOps column is num2; both WPF grids bind AvgCpuMs. Lite twin LocalDataService.FinOps.Workload.cs fixed the same way.	New FinOpsTopAvgRankingTests (wire row carries 2.34; source pins for the storage reader, the Lite twin, the three grids).
W5	Version Store used tempdb's startup size (sys.master_files) as the size denominator.	PvsStatsCollector on-prem text reads database_id 2 from tempdb.sys.database_files; other databases still read sys.master_files (a UNION ALL inside the same scalar OUTER APPLY, row count unchanged). Azure SQL DB text untouched. Read-only check: only sql2025 tempdb changed (64 MB became 12,352 MB). G1c: the first W5 commit left a doubled comma (PvsStatsCollector.cs:208) that made every on-prem run fail with "Incorrect syntax near ','"; fixed to one comma.	PvsStatsCollectorDefinitionTests, new PvsStatsCollectorSyntaxTests (lexical check of 5 variants; live SET PARSEONLY ON when DARLING_TEST_SQL is set).
W6	Default Trace rows repeated and could be lost: (1) a 3.33 ms datetime tick vs a datetime2 watermark re-stored the newest event on every poll (9 identical "Server Memory Change" rows); (2) every Object:Created/Altered/Deleted has subclass 0 (Begin) and 1 (Commit), so a DDL was stored twice (5,506 each on sql2022); (3) a strict > bound lost events in the watermark's own tick; (4) auto grow/shrink stalls carry their BEGIN time as StartTime but are written when they END, so a later stored event moved the watermark past them.	Shared collector (DefaultTraceEventsCollector.cs, Darling and Lite, no schema change): bound is ft.StartTime >= CONVERT(datetime, cutoff); rows at the watermark's tick are dropped only if their EventSequence is in state key watermark_tick_event_sequences ("<ticks>:<seq>,<seq>"); a row is also returned when it belongs to the running trace and ft.EventSequence > @last_event_sequence, tracked by state key trace_event_sequence ("<sys.traces.start_time ticks>:<newest EventSequence>"), so a SQL Server or trace restart reads the new trace from its first event and a changed path or identity reads the whole rollover set. DDL arm keeps ISNULL(ft.EventSubClass, 1) = 1 (Commit). Readers drop stored copies on a key of the row plus event_sequence (two real events with identical columns in one 3.33 ms tick stay two rows; a stored copy repeats the sequence and collapses; a NULL sequence keeps the old key): Darling DarlingDefaultTraceReader (a HashSet of row and sequence before the page limit), Lite GetDefaultTraceEventsAsync (SELECT DISTINCT with event_sequence), WPF ViewerDataService; viewer and Lite also dedup on filename and event_class, as the MCP reader does. Lanes B, G1b-c, G4a-b, H4b.	DefaultTraceEventsCollectorDefinitionTests, new DefaultTraceEventSequenceWatermarkTests (22; linked into Darling.Tests), reader tests in both products, plus one new case per reader (Darling MCP, viewer, Lite) that fails on the old key.
W7	renderLineChart never broke a line: a null value was skipped and its neighbours joined across hours with no samples.	charts.js: exported gapThresholdMs (3x the median interval of the series, floor 5 minutes, no break under 3 points) and splitAtGaps; line, area, stacked-area and the dual-axis overlay draw one polyline/polygon per run and a dot for a lone bucket after a gap. The threshold is per series (a 15-minute overlay over 1-minute rows is one line). A series with no regular cadence (the 75th percentile of its intervals is more than 3 times the 25th, as with clustered events) is never split (gapThresholdMs returns Infinity), so sparse event charts draw one line, not lone dots; a sampled series, even one missing a quarter of its samples, is tight and still breaks past the 5-minute floor. Stacked-bar untouched. Lanes E, G4a, H4b.	WebChartGapBehaviourTests (15, harness web-chart-gap-harness.mjs): per-series case, clustered events draw one line (line and area), a 1-minute series with a 7-hour silence still draws two.
W8	The Waits tab showed MCP-caller wording ("pass bucket_minutes (1-1440) for another width, or narrow hours_back ..."): aggregate_note was drawn as raw text and never ran through plainText.	Web layer only (MCP text unchanged): util.js noticeStrip/emptyStrip run text through plainText; new metricNote does the same for the four aggregate_note lines in server-tabs.js; FinOps finops-note lines in utilization.js and index-analysis.js too. New plain-text.js rules rewrite "pass bucket_minutes ... or", "raise limit ..." clauses, "Scope by server_name, or raise limit, to see the rest" (becomes "Select a single server to see the rest"), "pass database_name ..." (said as the page's database filter), "matched database_name 'x' with blocking_only ..." (becomes "matched database 'x' with blocking queries only with wait type 'y'"; collector_name and min_duration_ms too) and "healthy answer for this read rather than missing data" (becomes "a normal result, not missing data"). Later rules: "narrow the bucket to see it" becomes "narrow the time range to see it" (the pages have no bucket control); the Waits, Latch/Spinlock and Activity notes print display labels ("Wait ms/s", "Coll/s", "the top application, its connection count, ...") instead of field names; "24 hour(s)" becomes "24 hours" / "1 hour", and the (s) rule writes "ies" after a consonant plus y ("2 queries", "2 policies"). Lanes C, H1.	WebPlainTextTests (+2), WebWordingFixesTests (new, 3), WebWaitQueriesLinkBehaviourTests pin, TrendNotes_NameTheTimeRange_TheColumnLabels_AndAgreeInNumber.
W9	Memory tab "PEAK AT" showed peak_waiters_at (time of most waiters) beside the peak grant.	Darling DarlingMemoryGrantReader and Lite LocalDataService.MemoryGrants (semaphore and pool lenses) get a peak_granted_at column (DISTINCT ON the largest granted_memory_mb, newest on a tie, appended as the 12th column so no ordinal moved). get_resource_semaphore and get_memory_grants (Darling and Lite) carry peak_granted_at. The web grid shows "Peak waiters at" and "Peak granted at". Lane D.	McpLatestSnapshotStampTests (both products), WebAuditLaneDLabelTests.
W10	Latch/spinlock panels said "last 24 hours" while the columns were "MS/S" and "COLL/S".	Labels are "ms/s, latest interval" and "Coll/s, latest interval" (LATCH_COLUMNS, SPINLOCK_COLUMNS). Lane C.	WebWordingFixesTests.
W11	Long Collection Health and Job History text made rows up to 854 px tall.	New clampText() in util.js (full text stays in the node and tooltip; click/Enter opens the rest); column option clamp: true in panels.js cell(); CSS .clamp-text. Applied to seven Collection Health columns (Run time, Last Error, Note, Recovered, Output, Regression, Why not collected) and both Job message cells. Lane E.	WebClampCellBehaviourTests.
W12	Panels built from a row-capped read looked like a full window (system_health CPU; Active Queries returned 50 of 219 on SQL2016 and 50 of 255 on SQL2019).	windowFloorStrip also draws the row-capped note for line charts (only when it names oldest_shown_utc); new cappedReadNote covers a newest-first page with truncated: true and oldest_returned_collection_time; markPartialSubtitle replaces "last 24 hours" in the heading with "newest rows only, back to ". Client only. Lane E.	WebPartialWindowBehaviourTests, WebDataStartNote* pins.
W13	Collection Health said a normal zero (trace flags, running jobs with nothing to report) needed "a look".	New ZeroIsNormalCollectorNames in ServerHealthBands.cs (trace_flags, running_jobs, plan_correction, pg_replication_slots, pg_replication_stats) with its own FormatOutputFinding sentence ("Its source is empty unless something is enabled or running ... zero is a normal answer and needs no action"), chosen by a new optional isZeroNormal parameter; denials, faults and notes keep their own sentences first. Darling (DarlingDataReader.OutputFinding) and Lite (LocalDataService.CollectionHealth.cs). Lane C.	ZeroIsNormalCollectorTests (Darling and Lite), CollectionOutputBesideCostTests pin 6 to 7.
W14	Alert History Triage and Mute were not the first columns.	Triage and Mute are the first columns (the Select checkbox of an editing seat stays ahead). Lane D.	MuteRulesPageTests pin.
W15	Latch class / spinlock panel subtitle and picker label used a singular name.	plural field on NAMED_TRENDS (latch classes, spinlocks), used by the panel subtitle and the picker's "Top ..." label. Lane C.	WebWordingFixesTests.
W16	Blocking tab trend chart and table shared the title "Deadlocks"; the two empty states differed (and the chart's own empty text still showed the read's message).	Trend chart is "Deadlock trend", the table stays "Deadlocks". The Deadlock trend, Deadlocks table and Deadlock Severity panels, and the Overview tab's Deadlocks chart, all show "No deadlocks in this window." (DEADLOCKS_EMPTY_TEXT, emptyWins) only when the read's status is exactly empty (emptyHeadline in panels.js); a not_collected, unavailable or precondition answer keeps its own message and engine-gate line, so Azure SQL Database never reads as an all-clear. The collector-run count is dropped from the empty text. The blocking chart is titled "Blocked process report trend" with legend "Reports" and axis unit "reports" (Overview and Blocking tab; the count is unchanged, one point per report); when get_blocking_trend falls back to the DMV snapshot (source = "DMV snapshot") title, legend and axis follow it ("Blocking snapshot trend", "Snapshot rows") via bySource/applySourceWording. Lanes D, H1 (and H3a for the starter-dashboard template, see decisions).	WebAuditLaneDLabelTests, WebChartAtTimeBehaviourTests pin, EveryDeadlockEmptyState_OnTheBlockingTab_SaysTheOneSentence, ANotCollectedOrUnavailableDeadlockRead_KeepsItsOwnMessage_NotTheAllClear, ABlockingEmptyState_StillShowsTheReadsOwnMessage, TheBlockingTrendChart_SaysItCountsBlockedProcessReports, TheBlockingTrendChart_WordsItselfByTheCollectorThatAnswered, ChartIntegerTicksPin pins.
W17	Restart banner read "start time A -> B, server name X -> X" (an unchanged name listed as a change); plus an exact 24-hour start-time shift.	ServerEpoch.DescribeChange prints only the sides that moved (start time and/or server name, same test as IsNewEpoch, names case-insensitive), used by BaselineDiscontinuity and the ServerEpoch.ObserveInstance log line; if neither moved by that test both pairs print as before. The 24-hour shift is not confirmed (see decisions). Lane F.	Lite.Tests/BaselineDiscontinuityTests.
W18	Fixed by #5704.	-	-
W19	Labels did not match their values.	"HT ratio" is "Logical CPUs per socket". Fleet Threads tile detail is "N in use / M threads". FinOps Avg/P95/Max CPU are "... SQL Server CPU %". Fleet CPU chip says "Total CPU" when it shows total_cpu_percent (PostgreSQL stays "CPU"). Memory tab: "Plan cache (all cache stores)" (G1a corrected lane D: the value is the Plan Cache counter "Cache Pages" over all cache stores, read by MemoryStatsCollector, not a memory clerk) and "Cached plans size". "% of sweep" is "% of run budget". Significant Waits: "Duration" and "Signal duration". Lanes D, G1a.	WebAuditLaneDLabelTests, FinOpsTabUtilizationPageTests pin, WebClickthroughStaleStateTests ("Total CPU" chip).
W20	ISO UTC instants inside message text printed raw (System Events empty states "stored at 2026-10-12T19:12:15.2539650", the Triage status, the restart banner); the live check found more: Triage and Alert History bodies ("Incident Since: 2026-10-12 23:13:00Z", "Failed At", "Newest blocked process sample: 23:12:45 UTC"), Fleet Sweeps notes and Collection Health regression text.	localizeIsoText() rewrites ISO instants inside text to toLocaleString(); it takes a T or space separator, 0-7 fraction digits and a trailing Z, UTC or numeric offset (a space-separated stamp with no zone is a server's own clock and stays). Applied in emptyStrip, noticeStrip, the notebook Status cell and, by opt-in column flag localizeText, Collection Health notes, collector errors and regression text and the Fleet Sweeps Reason columns (the Sweeps liveness notes call the function). SQL, plan and query cells print as written. Alert bodies (Alert History, Triage, the notebook alert block, and the generic Alert History table through a localizeAlertDetail column opt-in on detail_text): localizeAlertDetail(text, newest_sample) rewrites only the labelled time lines (Incident Since, Started, Failed At, Snapshot, From, To, Window), never the Blocked Query, Query and Victim SQL lines; the time-of-day-only line ("Newest blocked process sample: 23:12:45 UTC, 5m 20s old") takes its date from the row's newest_sample.instant_utc, only when that instant's UTC time of day matches the line. Stored and emailed alert text is unchanged (Copy and CSV keep it; mute prefill parses it). Restart banner: start time A -> B (server clock); its own instant (at, store UTC) prints local. Lanes E, G1a, G4a, H1, H2.	WebPartialWindowBehaviourTests (+2), WebClampCellBehaviourTests (+2) (each shape in an opted-in plain and clamped cell, raw in a non-opted cell, a SQL cell unchanged, server-clock text unchanged, body lines, a wrong sample leaves the time-only line), TheAlertHistoryDetailColumn_LocalizesItsTimeLines_AndNeverTheSqlLines, Lite.Tests/BaselineDiscontinuityTests.
W21	(a) Job History "Last Success" is the job's current last success repeated on every run row. (b) Plan Corrections repeated each open recommendation every collection cycle.	(a) "Last Success" column removed from the web grid on the Job History page and the Activity tab (job-history.js, server-tabs.js); the MCP payload keeps the field. (b) Web only: DarlingPlanCorrectionReader.PlanCorrectionsCollapsedSql / GetCollapsedPlanCorrectionsAsync (window functions) return each recommendation's newest capture with first_seen and capture_count; the cap counts distinct recommendations, newest last-seen first. /api/read/get_plan_corrections passes collapse: true; web rows gain first_seen, last_seen, capture_count; the grid shows "First seen", "Last seen" (the old Collected), "Captures". The partition (ROW_NUMBER, MIN, COUNT) includes query_id, so two queries reusing a name such as PR_1 after a restart stay two rows; a collection_time tie picks score DESC NULLS LAST, regressed_plan_id DESC NULLS LAST, last_refresh DESC NULLS LAST. Lanes F, G1d.	JobHistoryPageTests, DarlingMcpPlanCorrectionToolsTests, live PlanCorrectionCollapseLiveTests (15 per-capture rows collapse to 3; reused name stays separate; deterministic tie).
W22	?hours=abc and ?hours=1.5 were read as the default window and answered 200 (QueryInt reads through ParseInt, which returns the default when int.TryParse fails).	DarlingWebEndpoints.NonIntegerParamRefusal checks every integer knob the dispatch reads (hours, hours_back, days_back, trend_hours_back, baseline_hours_back, limit, top, bucket_minutes, min_dop, worst_count) and answers the standard {"status":"invalid"} envelope (HTTP 400): "Invalid hours value 'abc'. Must be a whole number." It runs before any tool in the /api/read/* loop and at the top of /api/fleet. Absent or empty knobs take the tool's default; a readable out-of-range number stays the tool's own 400; /api/fleet, which has no tool behind it, refuses hours/hours_back outside 1..168 and a negative worst_count itself instead of clamping (review, Y1). ParseInt unchanged. The out-of-range refusal on the web names the parameter the caller sent: ?hours=-5 and ?hours=500 answer naming hours (NameWindowParameterTheCallerSent, web only; the MCP tool keeps hours_back, and a request that sent hours_back is untouched). Lanes F, H1.	DarlingWebEndpointsTests (4 new: 9 non-integer cases refused, 6 readable/absent not, both routes call the check first), WindowRefusal_* in DarlingWebEndpointsTests.

Also, from the live checks: the app shell's missing icon (a /favicon.ico 404) is fixed by dev's #5732, whose icon link this PR keeps after the dev merge; this PR's own data:, link was dropped.

Decisions and known limits

Blocking (W1/W2).

• W2 is inferred: the exact audit rows could not be reproduced. The off-by-one fits a second one-report chain beside the "(x42)" chain; the fix removes the cause (one list for value and card).
• A different reported wait of the same pair starts at least the 5 s threshold later, so the 2 s merge window cannot join two waits and is far above millisecond jitter. No self-blocking filter (that is [BUG] MCP tool audit on 3.9.0-nightly.20261006.515: 23 defects (blocking, spills, FinOps, store, tool payloads) #5633 M1).
• Existing thresholds meet waits now: the blocking alert threshold and the analysis BLOCKING_EVENTS grading pair (10, 50 events/hr peak) compare against waits, not reports. A wait that ends before the first 5 s report counts as before; a wait of N seconds added about N/5 and now adds 1. Counts fall most for long waits, so an alert set against the old per-report number fires less; lower the threshold to keep the old sensitivity. The (10, 50) pair was measured on reports and was NOT retuned (per the ruling no number changed).
• Shipped BlockingCountThreshold is 1 (DarlingConfig.cs:759, viewer settings 493), so a 5-minute wait (about 60 reports before, 1 wait now) fires the count alert before and after. The duration gate BlockingWaitSecondsThreshold ([FEATURE] Blocking alert: threshold on total blocked wait time (level-triggered), not just event count #1839) is OFF by default (0). The analysis BLOCKING_EVENTS grade no longer warns on one long wait: 60 reports in a 4 h bucket read 15/hr (above the 10 warning band) before, 1 wait reads 0.25/hr and grades nothing; its max_wait_time_ms/average metadata are unchanged.
• The alert read still caps at 200 report rows per pass, so a very large storm is still capped. The analysis peak bucket places a wait in the bucket of its latest report. The alert card now shows XE waits only when any XE exists (it used to add uncovered DMV rows).
• get_blocking, get_blocking_trend, get_blocking_stats and the Blocked Process Reports grids still count per-report rows on purpose (separate reads). The "Blocking (recent)" tile, get_server_summary "recent blocking count", the "Blocking Detected" label and the XAML threshold boxes were checked and left (none say "reports").
• G1d did not revert its fix to watch the new tests fail (by reading, the old shape falls back to DMV in the input the first tests use).

W3. The 20 s tolerance is the choice; records are 60 s apart. No dedup on the record id: the store has no column for it (that needs a Darling migration plus a Lite DuckDB change). Limits: a wall-clock step larger than 20 s between two polls can still repeat a record, and near-duplicates already stored are not cleaned up. Lane B did not revert the fix to watch the new test fail.

W4 / W9 / test gaps. Lane B did not run the PG-backed golden/live FinOps classes (no rig). Lane D did not run the Darling live PostgreSQL stamp test (no rig), so the PostgreSQL form of the peak_granted_at CTE was unexecuted by that lane; it read each defect from source (its browser call was refused). The collapsed Plan Corrections SQL ran on a rig with 15 rows only.

W5. The monitoring login must be able to read tempdb.sys.database_files (tempdb's guest user is enabled by default). PVS exists only on 2019+ (the collector is gated), so sql2016/2017 do not apply. The doubled-comma break never shipped, so it has no changelog line. DARLING_TEST_SQL is not set in CI, so there the live PARSEONLY test skips and the lexical-only check is the gate (PARSEONLY, not NOEXEC: syntax only, passes on versions lacking the DMV). No test project references ScriptDom, hence no real parser in CI.

W6. The first poll after upgrade has no tick or sequence state, so it can repeat the newest tick's rows once (readers collapse them). Absent, malformed or stale state drops nothing; a row with no EventSequence is kept. A failed state save repeats rows (collapsed by the readers' DISTINCT) and never loses one. Limits (G4b): (1) a state table that is reset or lost falls back to the StartTime-tick rule, as before this PR: a stall written late in the window between the last good poll and the reset is not recoverable; (2) the model assumes the trace flushes in EventSequence order (sql2022 showed no inversions; the StartTime floor covers an inversion at or after the watermark); (3) the trace-start comparison uses the server's local clock, so a SQL restart inside a daylight-saving fall-back hour could class an old-trace row as the running trace's; (4) a persistently failing state save repeats the same rows each poll until it succeeds. The late-stall loss predates this PR. Forcing @last_event_sequence NULL fails 7 of the 22 new sequence tests.

W8 / W10 / W13. Not checked on a live page: other pass <param> clauses in PG tool notes (full_text, include_defaults); other parameter phrases sit in MCP tool Descriptions, which pages do not show; hours_back, days_back and as_of were already covered by #5554. W10 says "latest interval" rather than "latest minute" (the collector's sample interval). W13 judged the other collectors from names and sources only; the AG states were already handled; pvs_stats, waiting_tasks, query_snapshots and job_history were left out; the new names are not in the event list (its doc keeps running_jobs out; a test requires the activity-sourced set disjoint from it). Coordinate with #5633 (helper-6) if it edits the same list.

W7 / W11 / W12. Stacked-bar is untouched by W7; G4a's per-series test did not demonstrate the old-code failure. W11: Copy and CSV read row data, untouched (grid-tools.js not edited). W12: server side left alone because get_active_queries is not in WebDataStartNote's closed lists (a server change with census pins).

W17. The exact 24-hour start-time shift is not confirmed and no code cause was found: the start time is sys.dm_os_sys_info.sqlserver_start_time read as datetime2 (WaitStatsCollector, CpuUtilizationCollector), stored with ToString("O"), parsed with RoundtripKind, compared by exact value in IsNewEpoch; nothing rounds or shifts it, so a +24h shift must be what SQL Server reported (for example a restart at the same time of day, or a VM clock change). The six servers on DARLING01 show unrelated start times, so the audited pair was not seen.

W20. Banner start times are labelled "(server clock)" and NOT written as ISO-8601 UTC: sqlserver_start_time is the target's LOCAL clock and the host does not know its offset (ServerEpoch.Stamp compares "as the target reported it, never converted"), so a "Z" would let the web shift it by the server's UTC offset and print a start time that never happened. Lite and the WPF viewers get the same text from BaselineDiscontinuities.Compose. A true UTC start time later needs the server's UTC offset in the carrier query (a SQL change in two collectors); not done.

W21. DEVIATION (decision for the coordinator): the brief said to fix MCP callers too; not done. get_plan_corrections documents "Rows recur per capture, not per distinct recommendation" in its head on both products, pinned by McpToolGuideHeadsSqlTail*, the tools-list byte budgets and the Lite twin's reader (which also feeds the desktop grids). The MCP tool and the Lite twin still return per-capture rows (a contract change with its own pins); last_success stays in the MCP payload.

W22. bucket_minutes is now refused up front for a non-integer; before, the trend reads that parse it with OptionalInt answered "Invalid value for parameter 'bucket_minutes'. Expected a number." and the heatmap silently used 5.

Final review and late fixes (H lanes).

• Deadlock all-clear (HIGH, fixed in H1): the three deadlock panels first showed "No deadlocks in this window." for any empty-looking read, so a not_collected (Azure SQL Database), unavailable or precondition answer read as an all-clear. Only status empty gets the sentence now.
• Lite server tab Blocking badge (MEDIUM, fixed in H4a): GetAlertCountsAsync counted reports (about 11 for a one-minute wait) beside a tile and alert that said 1; it counts waits with the same BlockingWaits.WaitsCteSql text as the Overview tile.
• Daily summary (LOW, fixed in H4a): both daily statements counted per report; they now read wait CTEs, bucketed on the day of the wait's longest report. The field blocking_events and the tool descriptions are unchanged (they already say "events").
• Anomaly detectors (LOW, a ruling): PgAnomalyDetector and Lite/Analysis/AnomalyDetector.cs keep counting reports because their baselines are report-based. get_blocking* reads and the Blocked Process Reports grids also keep per-report rows, as does the Blocking tab blocking trend chart (the live check saw 10-12 points a minute beside the tile's 1; H1 words the chart as a count of blocked process reports).
• Default Trace dedup (LOW, fixed in H4b): the three readers' key ignored event_sequence, so two real events with identical columns in one tick showed as one row; it is part of the key now.
• Custom Views (LOW, NOT fixed here, filed as Custom Views: default_trace_events measures count stored copies of an event twice #5738): the default_trace_events measure (deftrace_duration_us, MeasureCatalog.cs) counts a stored copy left by an older build twice. A fix needs a ROW_NUMBER wrapper on the raw route in ComposeCompiler.BuildFactRelation (key as the readers plus event_sequence) and live PostgreSQL checks of the plan and the dimension-filter pushdown.
• Sparse event series (LOW, fixed in H4b): see W7; a rule on the series covers every caller with no panel changes.
• Starter-dashboard Blocking trend template (H3a): its custom-view panel carries no source tag, so H1 left it with the report wording; it gets a wording true for both sources.
• Favicon: dev's Web UI smoke check for the Darling pages and Web Admin (#5701, phase 1) #5732 added an icon link to the app shell while this PR was open; the dev merge keeps Web UI smoke check for the Darling pages and Web Admin (#5701, phase 1) #5732's link and drops this PR's (no changelog line here). The sign-in page (inline HTML in DarlingWebHostService.cs, out of scope) still requests /favicon.ico.
• W20 (H2): Triage and Alert History bodies, Sweeps notes and regression text are localized on the web only; the stored and emailed text, and the generated AlertDetailText.SampleAgeLine (shared with emails and Lite), are unchanged. A space-separated stamp with no zone is left alone because it is a server's own clock. No SQL line is ever rewritten.
• Live gaps: W6, W7 and W21(b) had no live data, W2 was checked with one chain only, and the late fixes in W7, W8, W16, W20 and W22 are covered by tests until the re-check of the final head (see the live table).
• Blocking band unit (review, Y1/Y2): the health band's count tiers (5 and 20 per server-hour) were measured in blocked-process reports, so the fleet card, the Viewer summary, the daily summary (Viewer calendar, MCP, Lite) and the fleet sweep band on the report count (blocking_reports, not added to the MCP payload). Every displayed count, alert value and MCP count stays waits. The day reason line says "reports/hr" when its rate comes from the report count.
• DMV snapshot grouping (review): the 2 s window is proven for blocked-process reports only. On DMV snapshot rows it can merge two separate waits of one pair that start under 2 s apart; BlockingWaits.cs says so, and Blocking waits: DMV snapshot rows can merge two separate waits of the same pair #5743 tracks the fix.
• Default Trace subclass numbering (review): checked on sql2022's sys.trace_subclass_values for Object:Created, Deleted and Altered (46, 47, 164): 0 Begin, 1 Commit, 2 Rollback, so the collector keeps the Commit row.
• First full CI run (after the PR left draft): 22 slow-class pins failed, all expectations outdated by intended changes in this PR (W4, W11, W14, W16, W20, H1, H4a, H4b); X1-X3 updated them, and the darling CI filter now reaches the Lite test file Darling.Tests links in.

Changelog lines

SECTION: Fixed
ENTRY: - Blocking counts one event per wait, not one per blocked-process report ([#5705]) - the alert value, the fleet "Blocking N" tile, the analysis, Lite's server tab Blocking badge and the daily summary (calendar and MCP) counted one long wait about once per 5 seconds; they now count one event per blocked session, blocker and wait start, the alert value equals the sum of the card's "Blocking chain (xN)" counts, and SQL Server, Lite (DuckDB) and PostgreSQL agree on sub-millisecond starts and missing times.
ENTRY: - Blocking alerts no longer fire on DMV waits when the only blocked-process reports are in excluded databases ([#5705]) - the XE-or-DMV choice is made before the excluded-database filter.
ENTRY: - CPU samples are no longer stored twice ([#5705]) - the newest ring-buffer sample was re-stored because its recomputed timestamp drifted a few milliseconds; the collector now ignores a stamp within 20 seconds of the watermark.
ENTRY: - FinOps "Top Databases by Avg CPU / Execution" lists the busiest database ([#5705]) - the average was rounded to whole milliseconds before ranking, so close averages tied; it now ranks on the unrounded average and shows two decimals.
ENTRY: - Version Store shows tempdb's current size ([#5705]) - tempdb's size came from its startup size in sys.master_files; it now reads tempdb.sys.database_files.
ENTRY: - Default Trace events are no longer repeated ([#5705]) - the newest event was re-stored on every poll (datetime rounding) and each DDL was kept as a Begin and a Commit row; the collector keeps only Commit, and the web, MCP, Lite and Darling Viewer grids show each event once, including events stored twice by an older build, while two real events with identical columns in one trace tick stay two rows.
ENTRY: - Default Trace events written late are no longer lost ([#5705]) - an event in the same 3.33 ms tick as the newest stored event, and an auto grow/shrink stall (stamped with its begin time but written when it ends), were never read after a later event moved the watermark; the collector now also reads every trace event written after the last one it stored (by EventSequence, keyed to the trace's start time) and reads a new trace from its first event.
ENTRY: - Memory tab shows the time of the peak grant ([#5705]) - the peak memory grant now shows "Peak granted at"; the old time is labeled "Peak waiters at"; get_resource_semaphore and get_memory_grants return peak_granted_at (Darling and Lite).
ENTRY: - Alert History: Triage and Mute are the first columns ([#5705]).
ENTRY: - Deadlock panels and the blocking chart are labelled accurately ([#5705]) - the Blocking tab's trend chart is titled "Deadlock trend"; every deadlock empty state (Blocking and Overview tabs) says "No deadlocks in this window." only when the read truly found none (a not-collected, unavailable or precondition read keeps its own message); the blocking chart is titled "Blocked process report trend" with a "Reports" legend and axis (or "Blocking snapshot trend" when only DMV snapshots exist).
ENTRY: - Labels match their values ([#5705]) - logical CPUs per socket, threads in use, SQL Server CPU in FinOps, Total CPU on Fleet, plan cache (all cache stores), cached plans size, % of run budget, Significant Waits durations.
ENTRY: - A non-integer hours (or other count) answers 400 instead of silently using the default ([#5705]) - ?hours=abc and ?hours=1.5 were read as the default window and answered 200; they are refused like an out-of-range value, and a web ?hours= out of range names hours. /api/fleet refuses an out-of-range hours or a negative worst_count instead of clamping it.
ENTRY: - The restart banner no longer lists an unchanged server name as a change ([#5705]) - it names only the start time, the server name, or both, whichever moved; start times are labelled "(server clock)".
ENTRY: - Job History no longer shows a "Last Success" time later than the run's own time ([#5705]) - that column held the job's current last success on every run; it is removed.
ENTRY: - Plan Corrections shows one row per recommendation ([#5705]) - with first seen, last seen and a capture count, instead of repeating every collection; two recommendations that reuse a name for different queries stay separate.
ENTRY: - Web pages no longer show MCP-caller wording, and a normal zero is not "a look" ([#5705]) - notes such as "pass bucket_minutes" and "raise limit" are rewritten for the page, trend notes name the time range, show display labels and agree in number ("24 hours"), latch and spinlock rate columns say "latest interval", and Collection Health no longer says trace flags or running jobs with nothing to report need a look (Darling and Lite).
ENTRY: - Charts break at collection gaps, long text cells show on one line, capped panels say so, and times use one format ([#5705]) - web charts no longer draw a line across hours with no samples, while sparse event series draw one line instead of lone dots; long Collection Health and Job History text clips to one line and opens on a click; panels built from a row-capped read (system_health CPU, Active Queries) show a partial-window notice; ISO UTC times in messages, collector errors, Triage and Alert History bodies, Fleet Sweeps reasons and notes, and Collection Health regression text print in local time (stored and emailed text is unchanged), while SQL text in grids keeps its date literals.

REF:
[#5705]: #5705

Deploy and live checks

Deploys to DARLING01:

• G2 (21:29Z): commit 4c0b81991 (this branch merged with origin/dev) as a service-only overlay, service 3.10.0 (dll SHA256 prefix C7B16FA72329); the merge brought dev migrations V174 (config.service_version) and V175 (rowcount_sum/avg_rowcount), so the store went 173 to 175 (additive, idempotent). After the restart: 5 servers Online, 0 failed collectors; default_trace_events HEALTHY on all 5; pvs_stats succeeded on SQL2019/2022/2025 with errors=0 (W5 syntax fix proven); 0 ERROR/FATAL in the service log. G2 also added [Collection("live-postgres")] to BlockingWaitCountTests to fix LivePostgresCollectionHygieneTests.
• G5 (23:00Z, service started 23:01Z): commit 0112c4649, which the G6 lanes proved is the build served and checked W1-W22 on, about 23:10Z to 23:36Z.
• Late fix lanes H1, H2, H4a and H4b came after those checks; H3a deployed the final head (below).

Live results of G6a-d (W18 excluded; it is #5704):

Item	G6 result	Detail
W1	FIXED	One 187 s wait on sql2019 made 34 reports; fleet total_blocking_events 1, card 1, header "Blocking 1". The trend chart and report grid still show one point per report (kept on purpose). Lite badge and daily summary fixed later (H4a; daily summary confirmed live by H3a).
W2	FIXED	One alert, value 1, one chain. The 43 vs "(x42)" case needs two chains; only one was made, so it is untested live.
W3	FIXED	After the 23:01Z restart every 1-minute CPU bucket holds 1 sample on all 5 servers; before, 22-29 of 83-84 buckets held 2.
W4	FIXED	SQL2025 top by average matches top_by_avg, ranked on the decimal. The exact tie is not on this fleet.
W5	FIXED on SQL2025	tempdb 64 MB became 12,352 MB. SQL2019 and SQL2022 cannot show a difference (equal sizes); their collector ran after the H3a deploy.
W6	NO LIVE DATA	No schema DDL stored since 2026-10-07; the collector excludes tempdb. H3a then created a table and a database: one stored row per DDL.
W7	NO LIVE DATA	No series has a gap over 5 minutes (largest CPU gap 2 min).
W8	FIXED where reachable	Notes lost bucket_minutes/hours_back text; "narrow the bucket", raw field names and "24 hour(s)" remained and were fixed by H1 (confirmed by H3a).
W9, W10, W11, W12, W14, W15, W17, W19	FIXED	Peak waiters/granted columns differ; "latest interval" columns; Job History tallest row 29 px (58 rows over 120 px before) and Collection Health 29 px (7 before); partial-window notes on System Events and Active Queries; Triage and Mute visible without scrolling; no "classs"; banner "(server clock)" with no "server name X -> X"; relabelled values as in the table.
W13	FIXED	running_jobs and trace_flags (0 rows) say "zero is a normal answer"; "needs a look" remains on SQL2017 waiting_tasks, left out on purpose.
W16	PARTLY	Titles right, but the empty messages showed the reads' own wording; fixed by H1, confirmed live by H3a.
W20	PARTLY	Banner and grids local; raw UTC remained in the Triage incident block, Fleet Sweeps notes, Collection Health regression text and Alert History "Failed At"; fixed by H2 and H1 (H3a confirmed all but regression text, which had no data).
W21	(a) FIXED, (b) NO LIVE DATA	(a) no LAST SUCCESS column. (b) get_plan_corrections returned 0 recommendations over 7 days.
W22	FIXED	hours=abc and hours=1.5 answer 400 "Invalid hours value 'abc'. Must be a whole number." on get_wait_stats and /api/fleet; hours=2 200; hours=-5 and 500 still 400 (then naming hours_back; H1 made the web name hours, confirmed by H3a).

H3a: deploy and live re-check

Deployed 5ba497c15 (branch merged with dev, plus the starter-dashboard wording fix) to DARLING01 at 03:31Z. Store v176 -> v177 (dev's #5709 migration; compiled_schema_version 177). Collecting within a minute; served JS carries the new strings.

• a. W16: FIXED. Blocking tab: Deadlock trend, Deadlocks and Deadlock Severity all say "No deadlocks in this window." Chart "Blocked process report trend", legend "Reports", axis "reports", "Source: blocked-process-report". DMV wording: NO LIVE DATA. The separate Deadlock Graphs panel keeps the read's own "No deadlock XML available in the specified time range."
• b. Notes: FIXED. CPU/Memory/Waits/Activity say "narrow the time range" ("bucket" remains only as "one 1-minute bucket"). Waits show "Wait ms/s", "Coll/s"; Activity "the top application, its connection count...". Queries: "last 24 hours".
• c. W22: FIXED. hours=-5 and 500 answer 400 naming hours; hours=2 answers 200.
• d. Favicon: FIXED on app pages (<link rel=icon href="data:,">; no favicon request or 404 after reload).
• e. W20: FIXED for Triage (Incident since 9/28/2026, 12:32:08 AM), Fleet Sweeps (service started 10/12/2026, 11:31:34 PM) and Alert History (0 raw-UTC time lines; SQL lines unchanged). NO LIVE DATA: Collection Health regression text. NOT CHECKED: generic get_alert_history table detail column (needs a Custom View).
• f. H4a: FIXED. SQL2019 2026-10-12 blocking_events 1 (max 183857 ms), for the 187 s wait that wrote 36 reports.
• g. H4b: NO LIVE DATA. Only sparse series is that wait: 3 adjacent 1-minute points, a 1 px stub on a 24 h axis. Harness tests cover it.
• h. W5: FIXED. pvs_stats succeeded after deploy on SQL2019 (03:59Z), SQL2022 (03:58Z), SQL2025 (03:36Z). SQL2025 tempdb data size 12.1 GB = tempdb.sys.database_files (12,352 MB), not the 64 MB startup size. SQL2019/2022 sizes are equal either way.
• i. W6: FIXED. After the 03:45Z collector run, grid and get_default_trace_events show one Object:Created and one Object:Deleted for the table, plus one Object:Created for the database. w6_ddl_check dropped and confirmed gone.

New defects: none. Observation: the Overview tab's Deadlocks chart still says "No deadlock was recorded ... 286 collector runs DID execute" (W16 scope was the Blocking tab).

Tests: Darling Guard 1588, Census 547, Ratchet 15, touched classes 70; Lite Guard 498, Census 67; all pass, 0 warnings.

Tests

Test classes added: BlockingWaitCountTests, BlockingWaitDuckDbTests, FinOpsTopAvgRankingTests, PvsStatsCollectorSyntaxTests, DefaultTraceEventSequenceWatermarkTests (linked into Darling.Tests), WebAuditLaneDLabelTests, WebWordingFixesTests, ZeroIsNormalCollectorTests (Darling and Lite), WebChartGapBehaviourTests, WebClampCellBehaviourTests, WebPartialWindowBehaviourTests, PlanCorrectionCollapseLiveTests, plus new cases in the existing classes named in the table. The late lanes added cases to BlockingWaitDuckDbTests (+2), DailySummaryBlockingWaitsLiveTests (+3, live PostgreSQL; all 5 H4a cases fail on the old code, checked by reverting the three source files), WebPartialWindowBehaviourTests (+2), WebClampCellBehaviourTests (+2), WebChartGapBehaviourTests, DarlingWebEndpointsTests (WindowRefusal_*), one Default Trace reader case per product (each fails on the old key), and the H1 tests named in W8, W16 and W20. Stale pins were updated, none weakened. Each lane ran its targeted classes plus Census and Ratchet green; the last lanes (H1, H2, H4a, H4b) also ran Darling Stage=Guard 1584 passed, Darling Census/Ratchet 560 passed (1 skipped), Lite Stage=Guard 496 passed, Lite Census/Ratchet 67 passed; CI runs the rest.

Claude posting for Erik Darling. All four findings are answered. (1) Band unit: the health band reads blocked-process report counts again (fleet card, viewer summary, daily summary, fleet sweep) while displayed counts stay waits, in 03cdb22 and d47ce50. (2) /api/fleet: out-of-range hours/hours_back and a negative worst_count are refused with a 400, in 03cdb22. (3) DMV 2 s grouping: BlockingWaits.cs now says the window is proven for reports only; the fix is tracked in #5743. (4) EventSubClass: checked on sql2022 sys.trace_subclass_values for events 46/47/164 (0 Begin, 1 Commit, 2 Rollback), so the collector keeps Commit. Details in the PR body's Decisions section. Own CI run 38359021451 is green at 6e8fa8a.