Skip to content

Release checks: fixes from the morning walk of Lite, the Darling Viewer and the web dashboard - #5554

Merged
erikdarlingdata merged 51 commits into
devfrom
fix/release-walk-1008b
Oct 8, 2026
Merged

erikdarlingdata merged 51 commits into
devfrom
fix/release-walk-1008b

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Fixes found in the morning walk of Lite, the Darling Viewer and the web dashboard. Each finding group has its own section below.

Running Jobs shows only jobs that are running now

Finding. The Running Jobs tab (Lite and the Darling Viewer) and the get_running_jobs tool (Lite MCP, Darling MCP and the web page that calls it) listed a job collected weeks ago under "Currently Running SQL Agent Jobs" with a duration of 31h 2m. The column headers "Current Duration" and "% of Average" were also clipped at the default width.

Cause. The running-jobs collector writes no row when no job is running, and the read returned the newest snapshot ever written. After the job ended, every later run found nothing, so the old snapshot stayed "the latest" for good.

Fix.

  • Lite: Lite/Services/LocalDataService.RunningJobs.cs GetRunningJobsAsync. The collector's latest SUCCESS run in v_collection_log decides whether the newest snapshot is current. A SUCCESS run that stored rows means the newest snapshot is that run's; a SUCCESS run that stored none (the collector logs SUCCESS with zero rows when nothing runs) means no job is running, unless a snapshot newer than that log row exists (rows are stored before the log row is written). A store with no SUCCESS log row for the collector keeps the old behaviour. Failed runs (ERROR, PERMISSIONS) are ignored; the msdb banner already reports them. The Lite MCP tool calls the same method.
  • Darling: the same predicate, over collection_log, in ViewerDataService.RunningJobsSql (Viewer tab) and DarlingJobReader.RunningJobsSql (MCP tool and web page). The Darling service stamps its log row when the run ends, after the rows are stored, so the "snapshot newer than the log row" arm and the "rows stored" arm both matter there; Lite stamps the run's start.
  • Column headers: widths in Lite/Controls/ServerTab.xaml and ViewerServerTab.xaml, Running Jobs grid only: Current Duration 120 to 150, Avg Duration 110 to 130, P95 Duration 110 to 130, % of Average 100 to 125, Running Long 90 to 125, Successful Runs (30d) 140 to 190.
  • Not changed: the alert read GetAnomalousJobsAsync already skips a stale snapshot by age ([QUESTION/BUG] Lite can repeatedly alert on stale Long-Running Job snapshot? #1812), so it never alerted on the old snapshot.
  • An empty result shows the empty grid, as it did before for a server with no running jobs.

Tests.

  • Lite.Tests/LiteRunningJobsCurrentSnapshotTests.cs (new, DuckDB store): a snapshot 35 days older than the last SUCCESS run reads as no jobs; a current snapshot, a snapshot whose log row is not written yet, a snapshot followed only by failed runs, and a store with no run log still read their jobs; a server with no snapshot reads as none. RED: 1 of 6 failed with the fix reverted. GREEN: 6 of 6.
  • Darling/Darling.Tests/ViewerPerfmonRunningJobsTests.cs, class ViewerPerfmonRunningJobsLivePostgresTests (live PostgreSQL, three new tests): the 3-day-old snapshot case and the current-snapshot case (including a later ERROR run and another collector's run), the no-log and newer-than-log cases, and a pin that the Viewer and MCP queries are the same text. RED: 2 of 6 failed with the Viewer query reverted. GREEN: 6 of 6.
  • Final head: Lite.Tests *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* plus the touched classes: Total 242, Failed 0. Darling.Tests the same wildcards plus ViewerPerfmonRunningJobsLivePostgresTests, ViewerRunningJobsSqlTests, DarlingMcpJobToolsTests: Total 975, Failed 0 (against a local test PostgreSQL). Both builds: 0 warnings. The column widths are not covered by a test and were not looked at in a running app.

Web dashboard: the Updated time, Job History messages and the Databases popover

All changes are under Darling/PerformanceMonitor.Darling.Service/wwwroot. The web dashboard has no Lite twin (Lite has no web front end). The Darling Viewer is a WPF app and does not use these scripts.

1. The footer "Updated" time stayed at the first load on server, FinOps and Job History pages, and on SQL Server Queries, Wait Stats and Admin with auto-refresh on.

  • Cause: js/app.js drew the footer time from the sidebar's fleet read (generated_at), not from the page. The page's own reload, manual or automatic, never touched it. I could not reproduce a fleet read that fails to advance, so the page stamp does not depend on it.
  • Fix: notePageUpdated() in app.js stamps the footer when each page's render settles (first load, auto-refresh, "Reload this page now"), and at once for pages the poll never re-renders. A later fleet poll no longer pulls the stamp back; before the first page settles the fleet time still stands in.
  • Test: StatusBarBehaviourTests.TheUpdatedTime_FollowsThePagesOwnLatestLoad (the status-bar region of the real app.js under Node; status-bar-harness.mjs gains a pageUpdated scenario).

2. Job History: the Message column was about 90 px wide, rows grew 150-250 px, and a row click opened nothing.

  • Fix: js/pages/job-history.js: the Message cell is a block of readable width (24 to 40 rem) showing the first 3 lines (.jh-message in css/app.css), with the full text on hover. A click or Enter on a row opens a detail pane above the grid with the job, step, server, status, run time, duration and the whole message (preformatted, scrolls past 24 rem). Close empties it, a click that ends a text selection does not open it, and the 60 s rebuild reopens the same run. The server page's SQL Agent tab draws the same read, so its Message column (js/pages/server-tabs.js) gets the same clamp and hover text.
  • Test: JobHistoryPageTests.ARowClickOpensTheWholeMessage_AndTheCellClampsIt (real page in the Node harness: cell text, click, pane text, reopen after rebuild, empty message, Close; plus a check of the CSS clamp and the server-tab column).

3. Custom view 3 ("SQL2025 - HammerDB Resources & Top Queries") opens on "All servers (fleet)" and shows hashes.

  • What the product does: a custom view stores its default range and its template variables, and a server-dimension variable's default is the view's saved server (the editor already uses it to grey measures a server cannot collect). js/pages/views.js seedState always opened the picker on "All" and sent server "All" with every run, which beats the $server variable on the service side (ParseServerScope). So a view saved with a server read the whole fleet. That is a product defect.
  • Fix: new savedServerScope() in views.js: one server variable with a concrete default that the fleet lists (by name or display name, or a comma or semicolon list) becomes the opening scope in both the dashboard and the notebook path. "All", blank, a $other reference, several server variables or an unknown name stay on the whole fleet. A scope the reader picked this session is still remembered over it. I did not read or edit the stored view, so I cannot say whether view 3 declares such a variable: if it does not, "All servers" is its true saved state.
  • "Top statements" showing hashes: unchanged, as in Stale servers read as stale, Collection Stopped counts from the last real collection, and Wait Stats reads spinlocks once #5492 finding 14. The measure is the statement dimension, which falls back to the query hash only when the statement has no procedure name. The walk found host_object null for those hashes (the monitor's own recommendation query and an ad-hoc insert), so the hash is the right answer for them.
  • Test: new ViewSavedServerScopeTests (the real seeding code under Node: a name, a display name, a list, and six cases that stay on "All").

4. The Databases popover opened on Queries stayed open on Wait Stats.

  • Cause: the open flag in js/pages/database-filter.js lives at module scope on purpose, so the 60 s poll's rebuild keeps a popover the reader is using; a tab click rebuilt the head and the new control drew open too.
  • Fix: new closeDatabaseFilter(); renderServer in js/pages/server.js calls it for every render that is not the poll's (tab click, another server, deep link).
  • Test: WebDatabaseFilterPickerBehaviourTests.ThePopover_StaysOpenThroughThePoll_ButClosesOnANavigation (real server.js render: open after click, still open after a poll render, closed with aria-expanded false after a navigation render).

RED: with only the production files of this section reverted, the new tests fail: TheUpdatedTime_FollowsThePagesOwnLatestLoad, ARowClickOpensTheWholeMessage_AndTheCellClampsIt (the harness throws on the missing click handler, so the rest of that class fails with it), AViewSavedWithAServer_OpensOnThatServer (1 of 8 ViewSavedServerScopeTests; the six "stays on All" cases pass on the old code by design), ThePopover_StaysOpenThroughThePoll_ButClosesOnANavigation. With the status-bar region reverted the other StatusBarBehaviourTests also fail, because the harness now exposes notePageUpdated.
GREEN: StatusBarBehaviourTests, JobHistoryPageTests, ViewSavedServerScopeTests, WebDatabaseFilterPickerBehaviourTests after merging this branch and dev: Total 60, Failed 0. With *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard*, *Web*, *Views*, *ServerTabs*, *Notebook* and those four: Total 3199, Failed 0, Skipped 177. Darling.Tests builds with 0 warnings. Nothing was run in a browser; the pane, the clamp width and the popover were not looked at in a running app.

Lite's own queries top its lists, and the I/O latency chart picks files with reads

Finding (Query Store and Top Queries). After the earlier fix that kept Lite's plan-correction query off the top lists, Top Queries by Duration and Query Store by Duration still began with Lite's own reads on a quiet server: the per-database "SELECT '' WHERE EXISTS (... sys.database_query_store_options ...)" probe (one row per database) and the Query Store collector's own "SELECT query_id = qsq.query_id ... INTO #text_fetch" and "SELECT plan_id = b.plan_id ..." reads.

Cause. Lite and the Darling service skip any statement whose text contains PerformanceMonitorLite. The Query Store collector put that marker in its main statements, but not in the probe's inner statement (built into @sql and run in each database, so it is the statement the monitored server caches and Query Store records), nor in the plan and text fetch by id, nor in the two result reads of the probe. The existing marker census only looked at literals passed straight to sp_executesql and never built the by-id or backfill queries.

Fix. PerformanceMonitor.Collectors/QueryStoreCollector.cs: the marker is in the probe's inner SELECT, in the probe's two result reads, and in both statements of the plan fetch and of the text fetch (the probe's @sql is widened to nvarchar(1000) for the longer text). The same marker is now in the connect-time detection query and the start-time read (Lite/Services/ServerManager.cs, Darling/PerformanceMonitor.Darling.Service/DarlingServerConnector.cs) and in the failed-jobs and job-step reads shared by Lite and Darling (PerformanceMonitor.Alerting/FailedJobsQuery.cs, AgentJobStepQuery.cs). Darling's Query Store reads are the same shared collector, so they are fixed by the same lines. Darling's one other collector-side read, the oversized-plan fetch, already carried the marker.

Test. CollectorSelfQueryMarkerCensusTests (Lite.Tests) now follows a batch built into a variable and run with EXECUTE @proc @sql, checks the probe's inner batch, checks that every top-level SELECT of every Query Store query (enumeration, per item, backfill, plan fetch, text fetch, in Lite's and Darling's shape, on old, new and Azure targets) carries the marker, and checks the detection and job reads. DarlingSelfQueryMarkerTests (Darling.Tests) checks the same for Darling's shape and its detection query.
RED (production lines reverted): CollectorSelfQueryMarkerCensusTests failed 3 of 5 (the probe test, the existing inner-batch test now that it sees the probe, and the top-level SELECT test); with only the detection and job-read lines reverted the new connect-probe test failed 1 of 6 and DarlingSelfQueryMarkerTests failed 1 of 2. GREEN: 6 of 6 and 2 of 2.

Finding (I/O latency chart). The File I/O "Read Latency" chart and the Overview "I/O Latency ms" lane were blank for a server whose legend listed only log files.

Cause (confirmed from the SQL and a seed). The latency read kept the ten files with the most reads plus writes. A log file writes constantly and never reads, so on a server with ten or more busy logs the ten were all logs, the read chart held flat zero lines, and the Overview lane (the average read latency of the same rows) was zero. The Lite and Viewer chart code both take the series as the read gives them, so nothing downstream could recover the data files.

Fix. Lite/Services/LocalDataService.FileIo.cs and Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FileIo.cs: the ranking is now two lists, the ten busiest files by reads (files with reads only) for the read chart and the ten busiest by writes (files with writes only) for the write chart, and a file in either list is charted. A tie on one count goes to the file with more of the other. The saved database filter still applies before the ranking. The Darling MCP file I/O tools were checked and need no change: DarlingTrendReader and Lite's TrendBuckets already rank series by summed stall, not by operations, and return read and write latency per series.

Test. FileIoLatencyTopFilesTests (Lite.Tests, DuckDB): twelve log files with 100,000 writes and no reads, two small data files with reads; the data files must chart with the right read latency (8.0 and 5.0 ms) and ten of the logs must still chart their 2.0 ms writes. The same seed runs against PostgreSQL in ViewerFileIoBlockingLivePostgresTests. Existing statement pins were changed on purpose because the ranking text changed: the Lite empty-filter head literal in DesktopDatabaseFilterReadsTests, the top_files and ORDER BY asserts in ViewerFileIoBlockingSqlTests (now the read and write ranks, and the old ordering must be absent), and the filter-before-cut check in DesktopDatabaseFilterLiveTests.
RED (SQL reverted): FileIoLatencyTopFilesTests failed 1 of 2; the PostgreSQL seed failed 1 of 10 in ViewerFileIoBlockingLivePostgresTests. GREEN: 2 of 2 and 10 of 10.

Tests run on the final head (Lite.Tests and Darling.Tests build with 0 warnings): CollectorSelfQueryMarkerCensus 6/0, FileIoLatencyTopFiles 2/0, DesktopDatabaseFilterReads 18/0, QueryStoreCollectorDefinition 55/0, LiteMemoryFileIoTrendBucketing 13/0, FailedJobsQuery 5/0, ServerManagerDetectionQuery 3/0 (Lite.Tests, Total/Failed); in Lite.Tests *Census* 61/0, *Scrub* 25/0, *Guard* 136/0 (no *Ratchet* or *Hygiene* classes there). Darling.Tests against a throwaway local PostgreSQL: ViewerFileIoBlockingSqlTests 19/0, ViewerFileIoBlockingLivePostgresTests 10/0, DesktopDatabaseFilterLiveTests 5/0, DesktopDatabaseFilterSourcePinTests 3/0, DarlingSelfQueryMarkerTests 2/0, AgentJobStepQuery 19/0; *Census* 467/0, *Ratchet* 15/0, *Hygiene* 90/0, *Scrub* 120/0, *Guard* 338/0.

Done in the I/O latency lane section below. The Overview "I/O Latency ms" lane now weights read latency by reads.

Server tabs: empty states, whole headers and plain wording (Lite and the Darling Viewer)

Findings D12, D13, D14, D17 and D19 of the walk. Lite and the Viewer share this front end, so each fix is in both unless noted.

  • Trace Flags (D12). The grid showed only its headers when no flag was enabled. Cause: its message element only held the "this collector does not run here" note and stayed hidden otherwise. Fix: the element now carries "No trace flags are enabled on this server." and shows it when the grid has no rows; the not-collected note still replaces it where the collector cannot run (ServerTab.xaml and ServerTab.Refresh.cs; ViewerServerTab.xaml, ViewerServerTab.Filters.cs, and a keepsOwnEmptyText option on ShowEngineGapAsync in ViewerServerTab.EngineGaps.cs). The Running Jobs empty state was left alone.
  • Query Store Regressions (D13). Real cause, from ViewerDataService.QueryStoreRegressionsSql: each query is compared with the 7 days before the selected range. At Last 7 days that baseline lies wholly before the stored Query Store data (about a week), so nothing can be compared and the grid is correctly empty; Last 24 hours has six days of baseline. The query is right; the message was not. The grid now says: "No regressions to show. Each query is compared with the 7 days before the selected range, so a range that reaches back to the start of the stored Query Store data has nothing earlier to compare with." (QueryStoreRegressionsEmptyText in ViewerServerTab.Queries.cs, set in ViewerServerTab.Filters.cs). Lite has no Query Store Regressions grid (only the MCP tool reads that comparison), so there is no Lite twin; a test pins that.
  • Configuration headers (D14). The Server, Database, Database Scoped, Query Store, Automatic Tuning and Trace Flags grids had widths too narrow for bold headers with a filter button ("Configured Valu", "Dynami", "Advanc", "Compat Le", "Recovery Mod", "Read On", "Snapshot Isolati"). Every column in those grids is now at least 6 pixels per header character plus 50 (30 columns widened, none narrowed), in both XAML files; the Running Jobs tab got its widths the same way.
  • Index Analysis notes (D17). The banner quoted the proc's internals ("Stage 1 did not capture ...", "adversarial tests 9a/10a assert ...", "Rule 7/7.5/7.5b reproduces the proc's LITERAL behavior ..."). IndexCleanupAnalyzer.BuildNotes (shared by Lite, the Viewer, the web dashboard and the MCP tool) now says what is left out of the scripts and what to check, in plain words; the developer reasons moved into a code comment. No other user-facing string in the analyzer or its models had those words. The FinOpsIndexAnalysis/golden.json fixture holds two of the notes and was updated by hand to the new text.
  • Current Waits (D19). Both charts now have a note under them: "Points appear only when a collection found waiting or blocked tasks. A gap means nothing was waiting at that time." The data is right (a row is stored only when a task was waiting).

Tests (ServerTabEmptyStatesAndHeadersTests, new, and IndexCleanupAnalyzerTests.Notes_AreInPlainWords_WithNoDeveloperJargon). RED with the production lines reverted: 8 failed of 71 (ServerTabEmptyStatesAndHeadersTests plus IndexCleanupAnalyzerTests): Trace Flags x2, Configuration headers x2, Current Waits x2, Query Store Regressions, the notes test. GREEN: ServerTabEmptyStatesAndHeadersTests + IndexCleanupAnalyzerTests + ViewerAzureSqlDatabaseGapCoverageTests + FinOpsTabIndexAnalysisPageTests 143/0. Lite.Tests (builds with 0 warnings): AzureSqlDatabaseNotCollectedSurface, *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* 227/0. Darling.Tests (0 warnings): *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard*, *FinOpsIndexAnalysis* 1051 total, 0 failed after the new class got the Reads=Lite trait its guard requires (27 skipped are live-PostgreSQL classes). Not run: the live FinOpsIndexAnalysisGoldenLiveTests (needs PostgreSQL; its fixture was edited by hand to the new note text). Nothing was run in the apps; no screen was looked at, so the header widths come from the measured clipping (about 5.3 pixels a character plus 48), not a rendered check.

Web dashboard: ages, counts and plain wording

Eight findings from the walk of the web dashboard (and the web's custom views). Page code is under Darling/PerformanceMonitor.Darling.Service/wwwroot/js.

  • Offline fleet cards (W1). An offline server's card, its Collectors chip and its five n/a tiles said only "no recent collection". The fleet read looks back two days on purpose (DarlingFleetReader.LastCollectionWindow, kept so the read stays cheap), so a server dark for longer than that comes with no last-collection time and no exact age can be shown without scanning the whole collection log on every refresh. They now say "last collected more than 2 days ago" (pages/fleet.js, DARK_PAST_WINDOW). A server with a collection inside the window keeps its exact age ("last collected 5h ago"). The Viewer's card reads without that window and already shows the real time, so it has no twin change.
  • Availability Groups (W5). Two reporting servers that see one group showed "1 Groups" over two cards. The tile labels now follow the count ("1 Group", "2 Groups"), and when there are more cards than groups a line under the tiles says each card is one reporting server's view of a group (pages/ag.js).
  • Freeze Headroom on an offline server (W6). Its empty answer said "a target it cannot classify may not be a PostgreSQL one at all". It now says that an offline server collects nothing, then that the collector has not run yet or the store has not learned the server is PostgreSQL. The statement and wait reads carried the same sentence and got the same fix (DarlingMcpPgWraparoundTools.cs, DarlingMcpPgStatementTools.cs, DarlingMcpPgWaitTools.cs).
  • OS CPU by Query (W7). The page turned any answer containing the words "permanent engine capability gap" into "does not apply to this kind of server", but the extension-not-installed answer says "not a permanent engine capability gap", so a fixable state read as a permanent one. The match is now the gate sentence's own opening ("This is a permanent engine capability gap", plain-text.js), and that panel shows "pg_stat_kcache is not installed on this server ... Install the extension" when the server reports the extension missing (panels.js, pages/server-tabs.js). Servers where the data truly cannot apply keep the old line.
  • Repro script time (W8). The "Generated:" line used the clock of the machine that built the script (DateTime.Now), so for the web it was the service host's local time with no zone. It is now UTC and says "UTC" (PerformanceMonitor.PlanAnalysis/ReproScriptBuilder.cs). Lite and the Darling Viewer share this builder, so their scripts now also say UTC.
  • Index bloat panel (W11). "answered_only excluded every row: ..." now reads "Nothing to rank: ..." with the Index Bloat table above named as the place that gives each index's reason. Every other answered_only and the PostgreSQL "cannot classify" sentences are covered the same way (the page's plain-text pass and the three messages above).
  • Custom view 3 (W4b). The view has no server variable; each of its five panels carries its own server filter for one server. The service ANDs a panel's filters with the scope, so the view already read only that server while the picker said "All servers (fleet)", and picking another server made every panel empty. When every composed panel of a view filters to the same single server (and the server is in the fleet), the picker now opens on it, with a line saying the view sets it. Picking another server, or All servers, replaces the setting for every panel: the server filter is dropped from each panel and the other filters stay. A server variable still decides first, and a mixed set of panels behaves as before (pages/views.js, dashboards and notebooks). The stored view is untouched.

Tests. ReleaseWalkWordingTests (new, 9 tests: fleet card both ways, AG counts both ways, precondition vs gate sentence, answered-ranking text, kcache panel wiring, the three PostgreSQL messages, the Repro stamp) and nine new cases in ViewSavedServerScopeTests (16 total), run against the shipped page modules under Node. RED with only the production lines reverted: ReleaseWalkWordingTests 6 failed of 9 for the JS fixes and 1 of 9 for the Repro stamp; ViewSavedServerScopeTests failed throughout once the helpers were gone. GREEN: both classes 25/0. Two existing pins were updated on purpose: the AG tile label pin (RollupTileDescriptionTests, now countLabel(num, lbl)) and the table() signature pin (ServerPageTabsTests, one more trailing parameter). After the merge: *ReleaseWalkWording*, *ViewSavedServerScope*, *WebPlainText*, *RollupTileDescription*, *ServerPageTabs*, *Census*, *Ratchet*, *Hygiene*, *Guard* 969 total, 0 failed (28 skipped, live classes); *Web* + *Scrub* before the merge 2553 total, 1 failed (StatementFilterWebTests timing test, passes alone 14/0). Lite.Tests *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* 210/0. Both test projects build with 0 warnings. Not run: the live-PostgreSQL classes; nothing was run in a browser or an app.

Worth a second look. W1 gives a bound ("more than 2 days"), not the exact age; the exact age would need the unwindowed collection-log read that FleetLastCollectionSql avoids on purpose. W8 changes the shared builder, so desktop scripts say UTC now.

Saved plans and exports withhold statement text the same way the screens do

Finding. A saved plan file or a copied or exported statement could not be checked against the screens, and the code showed no filter call on those paths. Every path that writes statement text or plan XML out of the app was traced, in Lite, the Darling Viewer, the shared PerformanceMonitor.Ui project and the web pages. Two Lite reads were raw.

Cause. The collectors judge statement text and plans where they first enter a row (QuerySnapshotsCollector.ReadAsync and the other collector reads), so anything read from the store is already judged. Two Lite sources are not collected and had no judge of their own:

  • The Live Snapshot button (ServerTab.LiveSnapshot_Click / ReadLiveSnapshotRow) has its own SQL read. The statement text and both plans of every running request went into the grid raw. They then reached every Copy cell / Copy row / CSV of that grid and the .sqlplan that the row's Download buttons save.
  • "Get Actual Plan" re-runs the query and gets the plan back from the monitored server. Six call sites (the server tab, the Query Stats, Query Store and Procedure history windows, the wait drill-down window and the FinOps tab) opened that plan in the viewer, where Save writes it, without judging it.

Fix. Both go through the filter the screens already use. ReadLiveSnapshotRow takes one SensitiveStatements.Session per read (one shared budget) and judges the text (Text) and both plans (Xml). The six re-run sites wrap the result in await LivePlanDisplay.FilterAsync(await ActualPlanExecutor.ExecuteForActualPlanAsync(...)) (the filter runs on a thread-pool thread, never on the UI thread), the same wrapper the plan fetches use. A plan that names nothing comes back as the same instance, so unaffected plans are unchanged. A Download of a withheld Live Snapshot plan shows the existing "withheld" refusal and writes no file.

Every path, and where it is judged.

Path Source Judged at
Lite plan saves from the grids (Download, Estimated, Actual) stored snapshot plan collection (QuerySnapshotsCollector.ReadAsync)
Same, from the Live Snapshot rows live read ReadLiveSnapshotRow (new)
Lite plan fetched by hash, procedure, Query Store plan id or sql_handle (server tab, history windows, FinOps, copy/export menu) live plan cache read LivePlanDisplay.Filter at each call (existing census EveryLiveFetchCaller_...)
Lite "Get Actual Plan" re-run, then viewer Save live execution LivePlanDisplay.Filter at the six sites (new)
Lite history windows' Save .sqlplan live fetch above LivePlanDisplay.Filter, then WithheldPlanGuard.RefuseSave
Lite deadlock graph / blocked process report Save XML, Copy stored collection (scrub.Xml), then FileSaveHelper.SaveXmlToFile refuses a withheld marker
Copy cell / Copy row / Copy all / CSV of any Lite or Viewer grid (DataGridExport, ContextMenuHelper) the grid's rows the row's source above (stored rows judged at collection, Live Snapshot rows at the read)
Plan viewer Save (shared PlanViewerControl) the plan it was given the plan it was given: every caller above hands it a judged plan; a plan the user opens or pastes from their own file is their own text
Darling Viewer plan saves (three history windows, tab Download) stored collection; WithheldPlanGuard.RefuseSave before the write
Darling Viewer Active Queries live snapshot, plan by sql_handle, actual-plan re-run service commands QuerySnapshotsCollector.ReadAsync (scrub.Text, scrub.Xml) and DarlingWorker.PlanResultOutcome (SensitiveStatements.Xml) before the answer is stored
Web Plan / Repro / CSV downloads the page's own fetch of an MCP tool answer the host's statement filter on every tool answer (#5342, #5362); the page builds the file in the browser from that answer

The Darling Viewer has no twin of the two Lite defects: it never opens its own connection to a monitored server (pinned below), and its live reads come from the service, which judges them.

Tests.

  • StatementScrubLiveReadTests (Lite.Tests, new, 5 tests): a Live Snapshot row with a named statement and a plan with a secret-named literal reads as the marker in the text and in both plans, with none of the canary's secret needles left; a plain statement and plan are unchanged; empty rows stay empty; every ExecuteForActualPlanAsync call in Lite (6, in 6 files) sits inside LivePlanDisplay.Filter(await ...); the set of Lite files that open their own SqlConnection is exactly the known four, so a new live read fails the test until someone decides what it reads.
  • StatementScrubViewerSavePathsTests (Darling.Tests, new, 3 tests): no Viewer file contains new SqlConnection(; every Viewer .sqlplan save site asks WithheldPlanGuard.RefuseSave first; the service handlers that answer a live read (QuerySnapshotsCollector.ReadAsync, PlanResultOutcome) still judge the text and plans. These pin existing behavior, so they pass on the old code: there is no Viewer defect to make them fail.
  • RED with only the production lines reverted (the seven edited Lite files): StatementScrubLiveReadTests 3 failed of 5 (the Live Snapshot row, the re-run scan, the connection census). GREEN: 5/0.
  • Final head, after merging the branch and dev: Lite.Tests *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* 218 total, 0 failed (includes the new class and StatementScrubLivePlanDisplayTests); Darling.Tests the same filters 963 total, 0 failed, 60 skipped (live PostgreSQL classes, not run). Both projects build with 0 warnings. Nothing was run in an app or a browser.

Worth a second look.

  • A store row written before the filter existed is shown as stored on Lite and Viewer screens and saved as shown; only reads through the MCP and web filters judge old rows again. Judging every old row on the desktop screens would be a design change to every grid, so it is not part of this section.
  • The Live Snapshot row keeps its plan buttons for a withheld plan (the flags come from the in-row plan), so Download shows the refusal instead of hiding the button.

Server tab toolbar names, the Compare box, empty Running Jobs and screen-reader text (Lite and the Darling Viewer)

Findings V10a, V10b, D20, D21 and D22 of the walk.

  • Toolbar combos (V10a). The time range, Compare, auto-refresh interval and time display combos had an empty name for UI Automation (a tooltip is help text, not a name). Each now has AutomationProperties.Name in both ServerTab.xaml and ViewerServerTab.xaml: "Time range", "Compare to", "Auto-refresh interval", "Time display". The shared AccessibleNames class names tabs, headers, rows and cells by class handler; it does not reach a combo, so a plain XAML name is the pattern used (as FinOpsTab.xaml already does).
  • Compare is greyed (V10b). Off by design: Lite supports it on Overview and on the Queries sub-tabs Top Queries, Top Procedures and Query Store; the Viewer on those three sub-tabs only (its Overview has no comparison overlay). The old code did set a tooltip, but WPF does not show a tooltip on a disabled control. Fix: ToolTipService.ShowOnDisabled="True" on the box and a tooltip that names the tabs it works on (ServerTab.CompareUnavailableToolTip, ViewerServerTab.CompareUnavailableToolTip). A second defect in the Viewer: it updated the box only on a Queries sub-tab switch, so after a visit to Top Queries it stayed enabled on Wait Stats, File I/O and every other tab where it does nothing. It is now updated on every inner-tab switch too (IsComparisonSupported), and the reset to None no longer triggers a second load.
  • Running Jobs (D20). The empty grid had no words. Its message element now carries "No SQL Agent jobs are running." and shows it when the grid has no rows, through the same keepsOwnEmptyText step as Trace Flags (D12). It stays hidden when the login or collector cannot read msdb (the warning above the grid says why) and the not-collected note still replaces it where the collector does not run.
  • Plan viewer (D21). Operator boxes were plain Borders, which have no automation peer, so the tooltip was unreachable. They are now PlanNodeBorder (new, in PerformanceMonitor.Ui, shared by both apps): a group element named for the operator, with help text that gives the tooltip's facts as sentences (cost, rows, object, warnings). The on-screen tooltip is unchanged.
  • FinOps headers (D22). The "Detail" and "Est. Savings ($/mo)" columns of the findings grid now have a header tooltip saying what the column holds, in FinOpsTab.xaml of both apps.

Tests (ServerTabToolbarAndTooltipsTests, new, 22 cases). RED with the XAML, Running Jobs, plan-box and inner-tab-handler lines reverted: 10 failed of 22 (combo names x2, ShowOnDisabled x2, Running Jobs x2, FinOps headers x2, inner-tab update, plan box wiring); the new IsComparisonSupported matrix and the plan summary and peer cases test new code. GREEN: 22 of 22. Also green on the final head: ServerTabEmptyStatesAndHeadersTests 8/8; Darling.Tests *Census* 475 (0 failed), *Ratchet* 15, *Hygiene* 90, *Scrub* 123, *Guard* 338, ViewerScreenReader* 17; Lite.Tests *Census* 61, *Scrub* 25, *Guard* 136, *ScreenReader* 17, EngineGapNoteTests 22, AzureSqlDatabaseNotCollectedSurfaceTests 14, RefreshCoordinatorTests 20 (ScreenReaderNamesTests.ABlankCell_LosesItsBlankName_WhenItsTextFillsIn failed once on the first cold run and passed on every re-run, alone and in the class). Both test projects build with 0 warnings.

I/O latency lane weighted by reads, Long Queries empty text, Databases "Select All" and an empty Compare (Lite and the Darling Viewer)

I/O Latency ms lane (finishes the "Not done" note in the I/O section above). Finding: the Overview lane drew one point per time as the plain average of every charted file's read latency, so a log file with no reads counted as a 0 ms read and pulled the figure down. Cause: the latency read returned only each file's ratio, with no read count, so the lane could not tell which files had reads. Fix: Lite/Services/LocalDataService.FileIo.cs and Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.FileIo.cs now also return each bucket's read and write counts (Reads, Writes on the point), and the new PerformanceMonitor.Ui/IoLatencyWeighting.cs makes the lane's figure total stall over total reads across the files that had reads. Used by the Lite lane (current line and the Compare ghost line) and the Viewer lane. The write figure follows the same rule on Writes; the lane draws only reads. The web and MCP reads were checked: DarlingTrendReader and Lite TrendBuckets return per-series latency ratios and rank by summed stall, and the Lite baseline for this lane (BaselineProvider, delta_stall_read_ms / NULLIF(delta_reads, 0)) already skips rows without reads, so none needed a change. Test: FileIoLatencyTopFilesTests (new: weighted figure 6.875 ms where the plain average is under 2 ms; source pins that both lane controls use the weighting; a weighting unit test) and the PostgreSQL seed in ViewerFileIoBlockingLivePostgresTests (reads 500 on the data file, 0 on the logs, weighted 8.0 ms).

Long Queries shown empty in Lite while the Viewer listed runs. Cause: by design. Lite's long-query completion trace is opt-in and off by default (the laptop's collection_schedule.json has long_query_completions disabled), while the Viewer's Long Queries tab reads runs from its own collection. The warning banner already said the trace was off, but the grid itself said the generic "No data for the selected time range.", which read as a bug. Fix: PerformanceMonitor.Ui/LongQueriesEmptyText.cs; the grid's empty text in Lite/Controls/ServerTab.LongQueries.cs and ViewerServerTab.LongQueries.cs now follows the trace setting: off says the trace is off, so no completions are collected, and names the settings path and the long_query_completions Enabled box; on says there were no completions in the window. Test: ReleaseWalkFilterAndCompareTests (wording both ways, and a source pin on each tab).

Databases "Select All" gave "12 of 12" (a real 12-database filter). Fix: PerformanceMonitor.Ui/DatabaseFilterSelection.cs: ticking every listed database, with Select All or by hand, stores no filter (All), so databases collected later are included and the button reads "Databases: All". Both ServerTab.DatabaseFilter.cs and ViewerServerTab.DatabaseFilter.cs store through it. Test: rule tests (all ticked, some ticked, none, empty list) and a source pin on each tab.

Compare with no data in the comparison period drew no line and said nothing. Fix: in Lite the Overview lanes show a note on the CPU lane ("No comparison line: no data was collected for the same hours yesterday", or "last week", or "N days earlier") when every comparison read came back and every one was empty (CorrelatedTimelineLanesControl: ComparisonEmptyBanner, ComparisonHasNoData, ComparisonEmptyText); a failed read is not reported as "no data". The Viewer's Overview lanes have no comparison overlay (its Compare box drives the three Queries grids only), so the lane note has no Viewer twin. On the three comparison grids (Top Queries, Top Procedures, Query Store), in both apps, the "Comparing against baseline" banner now adds "no data was collected in that period, so there is nothing to compare against" when no row has any baseline executions (PerformanceMonitor.Ui/ComparisonBaselineNote.cs). Test: ReleaseWalkFilterAndCompareTests (decision and wording, banner text with and without baseline rows, source pins on the lane and on all three grid loads in each app).

RED (production lines reverted, tests kept). Lite.Tests FileIoLatencyTopFilesTests + ReleaseWalkFilterAndCompareTests: failed 10 of 22 (the weighted lane figure, both lane pins, both Select All pins, the lane banner pin, both comparison-grid pins, both Long Queries pins). Viewer PostgreSQL seed with the read count forced to 0: 1 of 1 failed. GREEN: the same 22 pass.

Tests run on the final head (Lite.Tests and Darling.Tests build with 0 warnings; Darling.Tests against a throwaway local PostgreSQL), Total/Failed: Lite.Tests *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard*, *ReleaseWalk*, *FileIoLatencyTopFiles* 240/0; Darling.Tests *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard*, *ViewerFileIoBlocking*, *DesktopDatabaseFilter* 1000/0. A Lite test class that reads a Darling path needs the Reads=Darling trait (DarlingReadsTraitGuardTests caught it; both classes now carry it). Nothing was run in the apps; the charts and popups were not looked at on a screen.

Not done. Nothing open in this section.

Web reads that fail say so in the log, an unknown server is named, and offline fleet cards show their real age

Findings. (W12) Two web reads on a server page answered HTTP 503 and the service log had no line for them. (W13) Running a Custom View panel for a server that does not exist returned a null row and no error. (W1b) Offline fleet cards said "last collected more than 2 days ago" because the fleet read looks back only 48 hours.

Cause and fix.

  • W12. Every read path that answers 503 that I could find already logged through DarlingWebFailureLog (a caught statement timeout, a tool-caught 57014, the backstop). I could not reproduce an unlogged 503 from the code, and I did not read the production service log. What the lines lacked was the server and the cause. DarlingWebFailureLog.RouteOf now names the server the read asked for, the exception line carries a cause (statement_timeout, or client_timeout_or_pool_wait for Npgsql's own wait, since the driver words both alike and exception text is never read), and the web backstop (DarlingWebHostService) now writes one warning (error for other 5xx) for any /api/* answer of 500 or more that no failure report covered, so a 503 can no longer leave the service log empty. /api/ping keeps its own deliberate 503. A per-request flag (BeginRequestTracking) stops it doubling a line a route already wrote. On the page, a failed read already renders a red strip with the message; the one silent case was a panel that hides itself when it has no rows (the "Analysis could not read these data families" card), which hid on an error as if the server had no problems. server-tabs.js hidesPanel now keeps it and shows the error. There is no automatic retry on the page, so there is no retry to surface.
  • W13. RunComposedPanelAsync (the one runner behind the MCP run_custom_view_panel and the web /api/compose/run) answers a new not-found outcome when the scope names only servers the registry does not know and the rows hold no value. The MCP tool answers status: not_found with "No server named 'x' is registered, and nothing is stored under that name."; the web route answers 404 with the same message. A removed server whose history is still stored keeps returning its rows (Custom Views and custom alert rules filter raw reads by server name, so every compressed chunk is read in full #5525), so the rule fires only when nothing is stored for the name in the window. Known gap: if the registry lookup itself faults, FindUnregisteredAsync reports every name unregistered, so a legitimate server with no data in the window would read as not found on that one run.
  • W1b. DarlingFleetReader.FleetOlderCollectionSql: for only the servers the 48-hour read found nothing for, one more read finds the newest collection before the window (a per-server LATERAL, ORDER BY collection_time DESC LIMIT 1, bounded above by the window start). It feeds the card's last_collection for display only; the Online/Offline decision still comes from the windowed read. A failed read logs a warning and leaves the old "more than 2 days ago" text. fleet.js already shows last_collection when present, so only its comment changed.

EXPLAIN on a local PostgreSQL (20,000 collection_log rows, two servers): Index Only Scan Backward using ..._idx_collection_log_time under an ordered ChunkAppend, startup chunk exclusion, Index Searches: 2, 6 shared buffers for both servers, older chunks "never executed". An index read, not a scan.

Tests, each RED then GREEN.

  • W12 log line: DarlingWebFailureHandlingTests (24 total, 0 failed). RED: with the backstop block disabled, A503TheRouteAnsweredWithoutReporting_StillWritesOneWarning_NamingTheEndpointAndTheServer fails (1 of 24). New: ping 503 stays unreported, a reported timeout still writes one line and names its cause, RouteOf names the server.
  • W12 page: ReleaseWalkWordingTests.AProblemListThatCouldNotBeRead_StaysOnThePageAndShowsTheError_... runs the shipped server-tabs.js under Node with every read answering 503. RED (fix reverted): the card's display is none and no error text. GREEN: shown with "The store took too long to answer this read." (11 total, 0 failed.)
  • W13: DarlingMcpCustomViewToolsLivePostgresTests (live, adds an unknown server -> not_found and a registered server still returns rows) RED 1 of 1 with the check disabled; ComposeUnknownServerMessageTests (3 total, 0 failed) covers the message, the stored-history and fleet cases, and the web 404.
  • W1b: DarkPastRetentionReadsOfflineLivePostgresTests now asserts the real last collection for a server dark five days, none for one that never collected, and the windowed value for a collecting one. RED 1 of 1 (expected 2026-10-03, got null) with the display line reverted. FleetReadsAreBoundedLivePostgresTests had pinned the old null for that same server; I changed that one assertion on purpose to the real age (13 total, 0 failed). ReleaseWalkWordingTests.ADarkCardThePageGetsALastCollectionFor_ShowsTheRealAge_... covers the page (12d ago).
  • Also run on the final head, 0 failed each: *Census* 475, *Ratchet* 15, *Hygiene* 90, *Scrub* 123, *Guard* 338, WebExceptionTextCensusTests 40, FleetCardTellsDark* 7, DarlingMcpCustomViewTools* 32. Build 0 warnings. Two of these first failed on my own change (a Guard member-range scan and the exception-text census, which rejected reading ex.Message for the pool-wait cause) and were fixed, not loosened. Not run: the full suite (CI decides) and Lite tests (no Lite file touched; the web dashboard and the service are Darling-only, Lite has no web front end or fleet read).

Not done. The production unlogged 503 itself is unreproduced; the new backstop line is the safety net for any path I missed.

Not done

Nothing open in this section.

Follow-up fixes (desktop)

  • File I/O charts (Lite and Viewer): fixed (69b9745). Each chart now draws its own ten files as the read gave them (the read chart the ten busiest by reads, the write chart the ten busiest by writes), through one shared FileIoChartFiles helper. The earlier cut of the combined list to one shared ten by summed latency is gone, so twelve busy log files can no longer push both data files off the Read Latency chart, and a write-only log file is no longer drawn as a flat 0 ms read line. The sentence above that said the charts took the series as the read gives them is now true. Tests: the 12 log files at 3 ms writes and 2 data files at 1 ms reads case (read chart shows the 2 data files), in Lite.Tests and Darling.Tests, plus a source pin on both chart methods.
  • Running Jobs freshness (Lite, Viewer, MCP tool; the web page reads the same tool): fixed (69b9745). A snapshot older than the alert read's long-running-job bound (AnomalousJobsResult.MaxSnapshotAge: three missed cycles at the server's effective running_jobs cadence, at least 10 minutes) is no longer listed. When nothing is listed for that reason the tab and the tool say collection is not current and give the time of the last good collection (unavailable status on the tool). A collector that keeps logging SUCCESS with no rows is current and gets no such note. The cadence comes from the schedule the app uses (Lite's schedule manager, the store's per-server then fleet override then the shipped default for the Viewer and service). The rule lives in one place, RunningJobsCurrency. Tests (RED shape: a run that stored rows followed by hours of ERROR runs, a switched-off collector, a store with no log rows): Lite.Tests LiteRunningJobsCurrentSnapshotTests and a live PostgreSQL test for the Viewer read and the MCP read. Two older tests encoded the old answer and were changed on purpose: a snapshot 30 minutes old followed by failed runs, and a 3-hour-old snapshot with no run log, now read as not current; the same cases at 8 minutes and 4 minutes still list the job. McpLatestSnapshotStampLivePostgresTests now seeds its running_jobs rows 2 minutes old instead of 2 hours.
  • r1 L3, Lite Running Jobs read error: fixed (69b9745). The Lite tab no longer reads through the query wrapper that turns a failure into an empty list. A failed read empties the grid and shows "The running jobs could not be read: ...". The Viewer twin did not show the old text on first load but kept whatever the last load said; it now does the same as Lite and still reports the failure in the status bar.
  • r1 L4, NULL and tie order of the File I/O ranking: fixed (69b9745). COALESCE(SUM(...), 0) in the ranking sums, NULLS LAST on every key, and a name tie-break in byte order (COLLATE "C" in PostgreSQL, DuckDB's own order) in the latency ranking and the throughput top 10, in both stores. Tests: a DuckDB case (11 tied files and a NULL-reads file) and the PostgreSQL twin, which fails on the old shape (the NULL-reads file takes read rank 1 there).
  • r1 L8, Databases Select All: fixed (this push, DatabaseFilterSelection.Stored). The filter now collapses to All only when the collected-name read succeeded and every collected name is ticked. It compares against the collected names, not the list on screen that also holds the sticky names. When the read fails or returns nothing, a narrow filter stays narrow. Lite and the Viewer both remember the collected names from the last popover load (_databaseFilterCollectedNames, null after a failed read). Tests (ReleaseWalkFilterAndCompareTests): sticky names only with a failed or empty read, and "every listed box" versus "every collected name".
  • s1 L1, statement filter off the UI thread: fixed (this push). Live Snapshot reads and judges all rows inside ServerTab.ReadLiveSnapshotRowsAsync on a thread-pool thread, one session (one budget) for the read. All 24 LivePlanDisplay.Filter(await ...) display sites (the six Get Actual Plan sites and the live plan fetches) now call await LivePlanDisplay.FilterAsync(await ...), which runs the same filter on a thread-pool thread. The button's busy state is unchanged and the withheld-on-timeout rule is the same call. Tests: the Live Snapshot read is driven from a dedicated thread and every row must be read on a different thread; FilterAsync gives the same answer as Filter; the two scans that pin every call site now require the awaited FilterAsync shape.
  • s1 L3, connection census: fixed (this push). StatementScrubLiveReadTests now scans all of Lite/ (not only Controls, Windows and Helpers). It found ten more files that open a connection to a monitored server, each listed with its reason: the drill-down plan fetcher (judged before analysis), the three FinOps reads (index, recommendation and server-property reads carry no statement text), the two plan fetches in LocalDataService (every caller wraps them), the scheduled collectors (judged at collection) and the connection test in ServerManager.

RED proofs, each by reverting only the production lines locally, running the new tests, then restoring (Lite.Tests failures / class total, Darling.Tests failures / class total):

  • File I/O charts (read chart ranked like the write chart, both chart methods back to the old cut): FileIoLatencyTopFilesTests 3 of 11 failed (helper case, Lite chart pin, Viewer chart pin); the PostgreSQL case in ViewerFileIoBlockingLivePostgresTests 1 of 31.
  • Running Jobs freshness (cutoff removed): LiteRunningJobsCurrentSnapshotTests 3 of 10; ViewerPerfmonRunningJobsLivePostgresTests 1 of 7 (the Viewer read and the tool's read are compared in one test).
  • Running Jobs read error (old query wrapper back): ReleaseWalkFilterAndCompareTests 2 of 22 (Lite and Viewer files).
  • File I/O NULL and tie order (NULLS LAST removed): FileIoLatencyTopFilesTests 1 of 11 (the statement pin; the DuckDB run itself already orders NULLs last by default, so the pin is what guards it); ViewerFileIoBlockingSqlTests 2 of 31.
  • Databases Select All (old rule): ReleaseWalkFilterAndCompareTests 3 of 22.
  • Statement filter off the UI thread (read loop inline): StatementScrubLiveReadTests 1 of 7.
    GREEN on the final head (merged with the branch and dev; both projects build with 0 warnings), Total / Failed. Lite.Tests: *StatementScrub* 32/0, *ReleaseWalk* 22/0, *FileIoLatencyTopFiles* 11/0, *LiteRunningJobs* 10/0, *Census* 61/0, *Scrub* 32/0, *Guard* 136/0 (Lite.Tests has no Ratchet or Hygiene class). Darling.Tests, against a local test PostgreSQL: *ViewerFileIoBlocking* 31/0, *ViewerRunningJobsSql* 17/0, *ViewerPerfmonRunningJobsLive* 7/0, *McpLatestSnapshotStamp* 40/0, *DesktopDatabaseFilter* 8/0, *Census* 475/0, *Ratchet* 15/0, *Hygiene* 90/0, *Scrub* 123/0, *Guard* 338/0. The merge with dev conflicted in two Viewer files (ViewerServerTab.QueriesComparison.cs, ViewerServerTab.LongQueries.cs); both sides are kept: dev's timed reads and parallel trace check, with the comparison banner note and the Long Queries empty text applied after them. A CA1859 warning in FileIoChartFiles (the helper's private method returned the interface) is fixed.

Not done (desktop)

Nothing open in this section.

Follow-up fixes (web and service)

  • Registry lookup failure no longer reads as "not registered": fixed (4c39746). ComposeServerScope.FindUnregisteredAsync reports its fault (onLookupFailed); the panel run then keeps the old empty answer instead of a not-found. The message now says "in this time range", since it only checks the window. Tests: ComposeUnknownServerMessageTests (a failed lookup is never unknown; a lookup against a dead store reports the fault and still answers every name).
  • A throw ahead of the old backstop now leaves one line: fixed (4c39746). One observer (Hosting/DarlingWebFailureObserver.cs) is registered first, ahead of response compression, the Host guard and the auth gate; the old backstop is folded into it, so a failed request still gets exactly one line (a route's own report ticks the per-request tracking). It is a class, so the Host guard stays the first app.Use lambda. Tests: a throw from a middleware behind the observer writes one Error line and answers 500; a source pin that the observer is registered before compression and every gate.
  • Older-collection read for a dark server cached per server for 15 minutes: fixed (4c39746). The value cannot change while the server stays dark; the entry is dropped when rows return, a failed read caches nothing. Live test: a second poll inside the window returns the cached value after its rows are deleted; one past the window reads again.
  • Views: scope note and seeding: fixed (3a697b3). The note says "your pick" only when the reader picked the server (the server picker sets it, and it is remembered with the scope); a scope from the view's own server variable says "this view's own setting". An explicit server variable, "All" and a reference included, decides first: the panels' server seeds the scope only when the view has no server variable. panelFilterServer returns null when a panel has any server filter besides the one eq. Node harness cases for each.
  • Footer "Updated" time: fixed (3a697b3). The page scheduler stamps it only when the render's reads settled without an error (readErrorCount in util.js); a failed poll keeps the last good time. It counts any failed read during the render, the sidebar's included. Harness scenario failedPoll.
  • s1 L2, offline-age read held two pooled connections: fixed (4c39746). The first read is closed before the second opens. Live test with a one-connection pool.
  • s1 L5, server text in 5xx lines: fixed (4c39746). The line names the server as the registry does when the read resolved one (the resolver notes it), and otherwise the request text passed through the diagnostics bundle's aliaser (addresses and secrets are rewritten; a partial or unregistered spelling cannot be, because the aliaser only knows registry names). Sanitize also turns Unicode format characters (category Cf, bidi controls included) into a dot. Tests for both.

RED/GREEN, by reverting only the production lines locally: the lookup, fault, route-name, format-character, cache and one-connection reverts failed 5 + 2 + 1 classes of tests (DarlingWebFailureHandlingTests 5 of 28, ComposeUnknownServerMessageTests 2 of 5, the live older-collection test 1 of 1) and passed when restored; the views and footer reverts failed 6 of 22 (ViewSavedServerScopeTests) and 1 of 8 (WebRenderSettleTests). The gate-throw observer test passes even with the catch-arm report removed, because the unreported-5xx line also writes one; the observer's existence is what the source pin and that test hold.
GREEN (Darling.Tests, local test PostgreSQL): DarlingWebFailureHandlingTests 28/0 failed, ComposeUnknownServerMessageTests 5/0, ComposeServerScopeByIdTests and FleetReadsAreBoundedByTheFleetTests with HostHeaderGuardTests 71/0, ViewSavedServerScopeTests 22/0, WebRenderSettleTests 8/0, Census 475/0, Ratchet 15/0, Hygiene 90/0, Scrub 123/0, Guard 338/0. Build 0 warnings. I did not merge origin/dev: it conflicts in a desktop Viewer file (ViewerServerTab.QueriesComparison.cs) outside this work.

Not done (web and service)

Nothing open in this section.

Follow-up fixes (round 2)

  • Guard stage (the STA gate census), fixed, 7ffd75f. The generic OnStaThread<T> in ServerTabToolbarAndTooltipsTests started an STA thread without using var staGate = WpfStaGate.Enter();. Added it the way AvailabilityGroupsTabRefreshTests does. -trait Stage=Guard: Darling.Tests 1411 total, 0 failed; Lite.Tests 446 total, 0 failed. No other STA thread start in this PR lacked its gate (the census passes in both projects).
  • M1, the footer's Updated time after a narrowed read, fixed, ecc2d0c. The read-error count moved in the fetch layer, so a read refused for a window wider than it takes, narrowed to its limit and then answered, stopped the footer on every poll although nothing failed. The count now moves in errorStrip, where the red strip is drawn (readErrorStrip still turns the window refusal into a notice and does not count it). That covers all the readWithinKeptHistory call sites at once. New harness case ANarrowedReadThatIsThenAnswered_DrawsNoRedStrip_AndLeavesTheReadErrorCountAlone (a narrowed and answered read: 0; a failed read with a red strip: 1; a refusal of the retry shown as a notice: 0). RED on the old util.js: 1 failed (of 34 in WebRangeKeptHistoryBehaviourTests plus ViewSavedServerScopeTests, which also showed the L3 pair).
  • L1, throttle for a failure before sign-in, fixed, 4259416. The observer is outermost, so its catch arm now also sees throws from compression, the Host guard and the auth gate for callers nobody has authenticated. A request marks PassedAuth on its tracking once it is past the auth gate (a small marker middleware right after it). Before that mark the line goes through DarlingHttpRefusalLog under a new PreAuthFailure key (once per source per ten minutes); the answer stays a 500. After the mark the line is written every time, as before. The Report doc says so. New test APreAuthThrowRepeated_LogsWithinTheThrottle_WhileAPostAuthThrowLogsEveryTime (five pre-auth throws: one Error line; three post-auth throws: three more). RED with the throttle removed: 1 failed of 29.
  • L2, pipeline-order comments, fixed, 4259416. The two comment blocks in ConfigurePipeline now give the real order (observer, compression, Host guard, auth, passed-auth marker, no-store, MapAll, static files) with the reason the observer adds no surface ahead of the Host guard.
  • L3, view scope note for an All or blank server setting, fixed, 2d5d669. The note now reads "Showing all servers, as this view's own setting says, instead of SQL2025, the server this view's panels are set to." New cases noteOnAllNotPicked and noteOnBlankNotPicked in the view-scope harness. RED on the old views.js: 2 failed.
  • L4, File I/O charts draw the read's own ten, fixed, 1e7ead4. The latency read (DuckDB in Lite, PostgreSQL in the Viewer) now returns two flag columns, read_rank <= 10 AND total_reads > 0 and write_rank <= 10 AND total_writes > 0, and FileIoChartFiles filters on them instead of ranking again on the points' counts (which leave out rows with no interval and broke ties on the joined "db.file" text). Both stores' tie-at-rank-10 tests (Ranking_NullReadsAndTiedFiles_KeepTheSameTenInEveryStore, run against the DuckDB store; FileIoLatency_NullReadsAndTiedFiles_KeepTheSameTenAsDuckDb_AgainstDevPostgres, run against a local test PostgreSQL) now also assert the chart's ten equal the read's ten, and the helper's twelve-logs case gains a file whose reads sit on interval-0 rows. The helper's signature changed, so the old tests cannot compile against the old shape; a revert-RED run is not meaningful here, and I did not run one.
  • L5, test gaps, fixed, 4259416 (gate-throw test); Running Jobs read-error pin, not replaced. AThrowFromAGateBehindTheObserver_WritesOneErrorLine_AndAnswers500 now also asserts the line carries InvalidOperationException and not "unreported", so it goes RED without the catch arm's Report (5 failed of 29 with the arm removed, including this one) and fails on a second line. The Running Jobs read-error check stays a source pin: the failure path is WPF code-behind (ShowEngineGap and the Viewer's catch in the tab's load method) with no helper that can be driven without a window, and extracting one only for the test would change shipped code for no user gain.

GREEN (Darling.Tests, local test PostgreSQL where noted): WebRangeKeptHistoryBehaviourTests, ViewSavedServerScopeTests, DarlingWebFailureHandlingTests, DarlingHttpRefusalLogTests, WebRenderSettleTests, ServerTabToolbarAndTooltipsTests 129/0 failed; ViewerFileIoBlocking* (with the test PostgreSQL) 31/0; Web*, ReleaseWalk*, StatusBar*, JobHistory* 1793/0 (97 skipped); Census, Ratchet, Hygiene, Scrub, Guard classes 965/0 (60 skipped); -trait Stage=Guard 1411/0. Lite.Tests: FileIoLatencyTopFilesTests with ReleaseWalkFilterAndCompareTests 33/0; Census, Ratchet, Hygiene, Scrub, Guard classes 222/0; -trait Stage=Guard 446/0. Both test projects build with 0 warnings.

CI fixes (four failing test classes)

Each one was treated as real; three were source-text pins that the PR's own edits reshaped, one was a real product gap. The new shape was pinned only after checking the guarantee still holds.

  • Lite OverviewLanesUtcFrameTests.EveryLaneX_IsTheSamplesOwnInstant_AndEveryGhostSeriesGoesThroughGhostX - cause: the I/O lane's Y became IoLatencyWeighting.Weighted(...) instead of g.Average(x => x.AvgReadLatencyMs). Guard: each lane's X is the sample's own instant, and every ghost series goes through GhostX. It still holds: the lane plots g.Key.ToOADate() (the group's instant), the ghost line is TimeWindows.GhostX(g.Key, days, zone), and the unchanged assertions (5 GhostX calls, none with .Add() pass. Fix: the pin now matches the new expression. Commit 88aa4e5.
  • Darling PgIndexBloatCoverageTests.TheWebDashboardPrintsTheCensusOnItsPopulatedPathToo - cause: table() in server-tabs.js gained a trailing extensionMissingLine argument, so the descriptor literal now ends span, noteKey, dbScope, extensionMissingLine }. Guard: the helper forwards noteKey so the panel's census note is rendered on the populated path, ahead of the empty and error branches. Still holds (the other assertions on renderer, getPath(res.data, desc.noteKey) and the panel's own noteKey window pass); the PR did not drop it. Fix: the pin matches the new literal. Commit 88aa4e5.
  • Darling OversizedPlanSweepRunRecordTests.TheFleetAggregatesOverTheCollectionLogExcludeTheSentinel (expected 2, actual 3) - cause: this PR added FleetOlderCollectionSql (the real last collection for an offline card), a third read over v_collection_log. It is per server and $1 comes from reads that already exclude the sentinel, so it could not return one, but it had no guard of its own, unlike the other two reads. Fix: product - it now carries AND s.server_id <> 0, the same predicate the other reads use; the test's count is 3 with a comment naming the three reads. Reverting only the SQL predicate fails the test (1 of 42); restored, 42 of 42 pass. Commit 88aa4e5.
  • Darling AlertNotebookRenderClientTests.Views_PanelOrError_SettlesExactlyOnce_SoABadPanelCellCannotHoldALimiterSlot - cause: the dashboard grid call is now panelOrError(panelUnderScope(p, state, panelServer), readSet, sourceSet, currentScope(), undefined, i). Guard: every panel settles its limiter callback exactly once. Still holds: panelOrError is unchanged (every error card goes through the settling fail; composed and read panels hand onSettled to their renderer), and the grid passes no callback, so panelUnderScope cannot take a slot. Fix: the pin matches the new call (text pin, no node harness case added). Commit 88aa4e5.

Checked on the head: the four classes (16, 42, 15 and 7 tests, 0 failed), plus *Fleet* (548, 0 failed), and *Census*, *Ratchet*, *Hygiene*, *Scrub*, *Guard* in Darling.Tests (477, 15, 90, 123, 338; 0 failed) and Lite.Tests (63, 0, 0, 32, 136; 0 failed; the Ratchet and Hygiene filters match no Lite class). Both projects build with 0 warnings. No full suite run.

SECTION: Fixed
ENTRY:

  • Running Jobs lists only jobs from the collector's latest run, and its column headers fit (#5554) - The Running Jobs tab in Lite and the Darling Viewer, and the get_running_jobs tool, listed the newest running-jobs snapshot ever written, so a job that ended weeks ago showed as running with a duration of more than a day. They now list jobs only when the collector's latest successful run found them, and show no jobs when that run found none. The Running Jobs column headers (for example "Current Duration" and "% of Average") are widened so they are no longer clipped. In the web dashboard, the footer "Updated" time now follows the page's own latest load, including auto-refresh and "Reload this page now". Job History messages get a readable, clamped column and open in full when the row is clicked. A custom view saved with a server opens on that server instead of the whole fleet. The Databases popover closes when you change tab. Lite's own Query Store probe, its plan and text fetches, its connect-time detection query and its failed-jobs and job-step reads no longer rank first in Top Queries by Duration and Query Store by Duration, in Lite and the Darling Viewer. The File I/O Read Latency chart and the Overview I/O Latency lane now chart the ten busiest files by reads, so a server whose busy log files outrank its data files no longer shows a blank read chart; the Write Latency chart keeps the ten busiest by writes. On the server tabs (Lite and the Darling Viewer), the Trace Flags grid says "No trace flags are enabled on this server." when it is empty, the Query Store Regressions grid explains that each query is compared with the 7 days before the range so a range as long as the stored history has nothing to compare with, the Configuration grid headers are no longer clipped, the Blocking Current Waits charts say points appear only when a collection found waiting or blocked tasks, and the Index Analysis notes use plain words instead of the stored procedure's internal rule names. On the web dashboard, an offline fleet card now says it last collected more than 2 days ago instead of only "no recent collection"; the Availability Groups tiles read "1 Group" and a line explains the cards; the Freeze Headroom, wait and statement empty answers and the index bloat ranking use plain words; the PostgreSQL "OS CPU by Query" panel says pg_stat_kcache is not installed instead of "does not apply to this kind of server"; the Repro script's "Generated:" time is UTC and says so (also in Lite and the Darling Viewer); and a custom view whose panels all filter to one server opens on that server, and picking another server replaces that setting. In Lite, the Live Snapshot grid and the Get Actual Plan re-run now run statement text and plans through the statement filter, so a withheld statement reads as withheld in the grid, in Copy and CSV exports of it, and in a plan file saved from it. On the server tab (Lite and the Darling Viewer), the time range, Compare, auto-refresh and time display boxes now have names a screen reader can read, the greyed Compare box shows a tooltip naming the tabs it works on (and the Viewer now greys it on every tab where it does nothing), the Running Jobs tab says "No SQL Agent jobs are running." when empty, plan operators expose their cost, rows and warnings to a screen reader, and the FinOps "Detail" and "Est. Savings" headers have tooltips. In Lite and the Darling Viewer, the Overview "I/O Latency ms" lane now averages read latency weighted by reads, so log files with no reads no longer pull it down; the Long Queries grid says the trace is off (and where to turn it on) instead of the generic "No data"; the Databases filter's "Select All" now means All (no filter) instead of a fixed list of every database; and a Compare period with no data says so, on the Overview lanes in Lite and on the Top Queries, Top Procedures and Query Store comparison banners in both apps. On the Running Jobs tab (Lite and the Darling Viewer) and in the get_running_jobs tool, a snapshot older than the long-running-job alert's freshness bound (three missed collection cycles, at least 10 minutes) is no longer listed as running; the tab and the tool say collection is not current and give the time of the last good collection, and a failed read shows its error instead of "No SQL Agent jobs are running." The File I/O Read Latency and Write Latency charts each draw their own ten files, and the file ranking orders NULLs and ties the same way in the DuckDB and PostgreSQL stores. In Lite, the statement filter on the Live Snapshot and the plan re-runs no longer runs on the UI thread, and the Databases filter keeps a filter for databases that are not collected yet when the list of collected names cannot be read. Failed web dashboard reads now log the server and cause, any 5xx answer without a report of its own gets a log line, and a server-tab problem list that cannot load shows its error instead of disappearing. Running a Custom View panel for an unknown server answers not_found (404 on the web) instead of a null row. Offline fleet cards show the real time of the last collection instead of "more than 2 days ago". Web dashboard follow-ups: a failed registry lookup no longer reports a real server as not registered, every failed request leaves one service-log line (including a throw in the sign-in or compression steps), a view's scope note says "your pick" only when the reader picked, and the footer keeps the last good Updated time when a poll fails. A failure before sign-in repeated from one address is logged once per ten minutes, a view's note says all servers when its own server setting is All or blank, and the File I/O latency charts draw exactly the ten files the read ranked.

Re-cut v3.8.0: Velopack upload hardening (#3521) to main
CI: the full-tier review runs up to 120 turns (dev to main)
…rigin/dev' into worktree-agent-af1afee1d8c064628
… read chart picks files with reads

Lite's Query Store database probe, the plan and text fetches by id and the probe's result reads carried no PerformanceMonitorLite marker, so they ranked first by duration in Top Queries and Query Store on a quiet server. The connect-time detection query, the start-time read and the failed-jobs and job-step reads carry it now too.

The File I/O latency charts and the Overview I/O Latency lane ranked files by reads plus writes, which kept only log files on a server whose logs carry the most operations. The ten busiest files by reads feed the read chart and the ten busiest by writes feed the write chart, in Lite and the Darling Viewer.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 8, 2026 12:12
… empty, Configuration headers fit, Index Analysis notes in plain words, Current Waits sparse-data note
…ere it works, Running Jobs says when nothing runs, plan operators and two FinOps headers are readable by a screen reader
… the trace switch, Databases Select All means All, empty Compare says so
…ver is not_found, and offline fleet cards show their real last collection
…, no not_found on a failed lookup, and a cached single-connection offline-age read
… plan sites; Databases Select All means All only when every collected name is ticked; the connection census scans all of Lite
…ext; File I/O chart file picker returns List (CA1859)
… auth; pipeline-order comments match the code; gate-throw test pins the catch arm
…ode, and guard the older-collection read against the sweep's sentinel id
@erikdarlingdata
erikdarlingdata merged commit fd80b59 into dev Oct 8, 2026
23 of 24 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/release-walk-1008b branch October 8, 2026 16:45
erikdarlingdata added a commit that referenced this pull request Oct 8, 2026
…er and the web dashboard (#5554)

Fixes found in the morning walk of Lite, the Darling Viewer and the web dashboard.

Running Jobs. The tab in Lite and the Viewer, and the get_running_jobs tool, list only jobs the collector's latest successful run found. A snapshot older than the long-running-job alert's bound is not listed. That bound is three missed cycles at the server's running_jobs cadence, at least 10 minutes. When that hides the jobs, the tab and the tool say collection is not current. They give the time of the last good collection.

Running Jobs read errors. A failed read shows "The running jobs could not be read" instead of "No SQL Agent jobs are running." The empty grid says so in words. The column headers are wide enough to read.

File I/O. The Read Latency chart no longer goes blank on a server whose busiest files are logs. Each chart draws its own ten files, exactly as the read ranked them. The read says which ten each file is in. The read chart takes the ten busiest by reads and the write chart the ten busiest by writes. The ranking puts NULLs last and breaks ties by file name the same way in the DuckDB and PostgreSQL stores.

Overview I/O Latency lane. It weights read latency by reads.

Lite's own queries no longer top Top Queries by Duration or Query Store by Duration. That covers its Query Store probe, its plan and text fetches, its connect-time detection query and its failed-jobs and job-step reads.

Saved plans and exports. The Live Snapshot button and the six Get Actual Plan sites judge statement text and plans with the same statement filter as the screens. The filter runs off the UI thread. A plan the filter withholds is refused on save. The test that lists every place Lite opens its own server connection now covers all of Lite, with a reason for each file.

Databases filter. Select All, or ticking every collected database, means All. Databases collected later are then included. A filter for databases that are not collected yet stays a filter when the read of collected names fails.

Server tabs in Lite and the Viewer. Empty Trace Flags, Running Jobs and Query Store Regressions grids say why they are empty. Configuration and Index Analysis headers and notes read in plain words.

Screen readers and Compare. The toolbar combos and plan operators have names for screen readers. The disabled Compare box says where it works. An empty comparison period says so. The Repro script's Generated line is in UTC.

Web dashboard, pages. The footer Updated time follows the page's own latest load. It keeps its last good value when a poll fails. A read narrowed to its limit and then answered is not a failure.

Job History and views. Job History messages are readable and open in full. A saved view opens on its server and says whose pick the scope is. It says all servers when its own setting is All. The Databases popover closes on a tab change. Offline fleet cards, Availability Group counts and several empty-state sentences read correctly.

Web dashboard, service. A failed web request leaves exactly one service-log line, and it names the server and the cause. A failure before sign-in from one address is logged once per ten minutes. An unknown server is reported as not found. A registry lookup that fails no longer reads as "not registered". The offline-age read is cached and holds one pooled connection at a time.
erikdarlingdata added a commit that referenced this pull request Oct 8, 2026
Adds the CHANGELOG entries for the PRs merged after the second splice. They cover the Viewer query tab loads (#5556), role provisioning under a lock (#5566) and the Query Store interval purge (#5572). They also cover Recommendations skipping secondary copies (#5564) and the release-walk fixes (#5554). The release-walk fixes get seven new entries, one per topic, written from the squash body. Each full entry goes into docs/changelog/3.10.md and its title-only line into CHANGELOG.md, with the archive census updated.

Changelog entries for the 3.10.0 gate PRs merged after the second splice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant