Repository navigation
[BUG] Web UI audit on 3.9.0-nightly.20261006.515: 22 defects (blocking counts, CPU duplicates, FinOps, charts, UI wording) #5644
Description
Component
Darling
Performance Monitor Version
3.9.0
SQL Server Version
All of them
Windows Version
All of them
Describe the Bug
Web UI audit of a live Darling install. This issue lists 22 web defects (W1 to W22), and the MCP defects that also show on the web.
Build: 3.9.0-nightly.20261006.515, with the web listener on the LAN. Date: 2026-10-09. A separate MCP audit found 23
defects. They are referred to below as M1 to M23.
How the audit ran
- Pages: a headless Chrome on Windows, with its own throwaway profile, driven over the DevTools protocol by
a small script. It opened 42 hash routes:- the top-level pages
- all 3 Admin tabs
- all 12 server tabs for one test instance, plus one overview for a second server
- all 12 FinOps tabs
- one alert triage deep link
- the three create forms, opened but never saved
- Data saved for each page: a full-page screenshot, the visible text, every
/apicall, console errors,
exceptions and failed requests. Nothing was clicked, so nothing was saved. - Positive control: a missing page, an unknown server and a missing view. The driver caught every 404 and 400
with its console error, so an empty error list means no errors. - Visual review: four reviewers read 94 screenshot tiles. I checked every finding below against the page text,
a DOM measurement, the store's API, the code or the SQL Server itself. Claims I could not confirm are left out. - Web API: the web-only routes were called with curl and a session cookie.
- Not tested: the 21 routes that write.
/api/server-databasesreturns 404 because it reacheddevthe same
morning the nightly was built.
Result at a glance
- All 42 pages loaded in 3.7 to 5.7 seconds, with no console errors, no exceptions and no failed requests.
- No page showed
NaN,undefined,null,Invalid Dateor[object Object]. - Every wide grid has a working horizontal scroller. Columns past the right edge can be reached, so they are not
listed as defects, apart from W14.
New defects
W1. A blocking "event" is one blocked-process report, so one long wait counts many times
On the test day, one instance showed "Blocking 43", and alerts fired with values 37 and 43. Grouping the 43 reports by
blocked session, blocker and batch start gives only 12 distinct waits:
| Blocked session | Blocker | Reports | Wait time across the reports |
|---|---|---|---|
| 102 | 100 | 11 | 5.9 s rising to 56.2 s |
| 102 | 100 | 9 | 6.9 s rising to 52.0 s |
| 102 | 100 | 9 | 7.1 s rising to 57.2 s |
| 102 | 100 | 6 | 6.3 s rising to 31.4 s |
The blocked process threshold is 5 seconds, so SQL Server reports a still-blocked session every 5 seconds. Every
report is counted as a new event, in the Blocking tile, the alert value and the analysis. A one-minute wait counts
as about 11 events. This is separate from M1, which is about a parallel query that blocks itself.
Expected: count one wait as one event (key on blocked session, blocker and wait start), and report its longest
wait time.
W2. The alert value and the alert detail differ by one
On Alert History, the value is one higher than the chain count in the detail for two rows:
| Time | Value | Detail |
|---|---|---|
| 10/9 9:19:15 AM | 43.0 | "Blocking chain (x42)" |
| 10/9 9:13:56 AM | 37.0 | "Blocking chain (x36)" |
The other blocking rows match, for example 58.0 with "(x58)".
W3. The CPU collector stores near-duplicate samples
sys.dm_os_ring_buffers on the test instance had exactly 60 scheduler-monitor records in the last hour, 60
seconds apart. The store has 82 to 87 samples in each full hour. In the last 61 minutes, 28 minutes have two
samples. The FinOps page shows this too: "1,593 CPU samples (24h)" on a server that was collected for less than
24 hours.
Cause: PerformanceMonitor.Collectors/CpuUtilizationCollector.cs (about line 179) re-reads the ring buffer
each poll. It computes sample_time from ms_ticks and SYSDATETIME(), then deduplicates on that computed value.
The comment there says that "a physical entry has no stable key". But each scheduler-monitor record carries a
Record id attribute, which is stable.
Expected: deduplicate on the record id. Duplicates inflate sample counts and percentiles, and they give extra
weight to the minutes that happen to be duplicated.
W4. FinOps "Top Databases by Avg CPU / Execution" leaves out the top database
On the FinOps page for one server, the busiest user database used 2,498,148 ms of CPU over 1,066,691 executions, which is
2.3 ms per execution. The "Avg CPU / Execution" table does not list it. It does list databases at 1.95 ms and
1.56 ms.
Cause: DarlingFinOpsDatabaseResourcesReader.GetTopResourceConsumersAsync (about line 366) converts
avg_cpu_ms with Convert.ToInt64 before ranking. 2.3, 1.95 and 1.56 all become 2. The tie then breaks on
database name, and the top 5 cut drops the busiest database.
Expected: rank on the unrounded average and round only for display.
W5. Version store panel shows tempdb's startup size, not its current size
The Version Store tab shows tempdb's "Database data size" as 32.0 GB. The Database Sizes tab shows 8 data files of
5.5 GB, which is 44 GB. On the server, sys.master_files gives 32,768 MB for tempdb, which is its startup size,
and tempdb.sys.database_files gives 45,056 MB, which is its current size.
Cause: PerformanceMonitor.Collectors/PvsStatsCollector.cs takes the size denominator from sys.master_files.
Expected: read tempdb's size from tempdb.sys.database_files.
W6. Default Trace shows every DDL event twice
On System Events, every DDL row appears twice with identical values. 58 of the 100 rows returned by MCP
get_default_trace_events are exact duplicates. On the server, the default trace has 3,280 Object:Created rows
with subclass 0 and 3,278 with subclass 1 in the last 24 hours. SQL Server writes one row when the operation
begins and one when it commits. The payload has no subclass column, so the two rows look the same. The duplicates
fill the row limit, and the grid covers only about 18 minutes.
Expected: keep only one subclass for object events, or return event_subclass.
W7. Charts draw straight lines across collection gaps
The service host is off every night. The CPU chart on the CPU tab draws a straight diagonal from about 11 PM to
6:30 AM, when no data was collected. The same happens in the Blocking charts, the tempdb write-latency chart, and
the second server's memory chart. On that memory chart, a restart shows as a smooth slide from about 98 GB to about
5 GB. A reader cannot tell the gap from real data.
Expected: break the line wherever no samples exist.
W8. Messages written for API callers appear in the UI
The pages show tool messages as written for an MCP caller. Examples:
- "see hints.captures for which collectors ran and when" (Overview, Blocking, second server's Overview)
- "pass bucket_minutes (1-1440) for another width, or narrow hours_back for finer points" (Waits, CPU, Memory,
Activity) - "widen hours_back to find the most recent samples" (Queries)
- "raise limit to see more" (FinOps Locking, which has no limit control)
- "in the last 1 hour(s) matched blocking_only" (Triage)
- "this is the healthy answer for this read" (System Events)
Expected: the web page should give its own wording, or name the page control that does the same thing.
W9. "PEAK AT" shows the wrong time for the peak memory grant
On the Memory tab, the peak granted memory of 86.5 GB has a "PEAK AT" time that is the newest snapshot. At that
snapshot, 0 MB was granted. The MCP payload has only peak_waiters_at, the time of the most waiters. With zero
waiters, that is the newest snapshot. The page shows it next to the peak grant.
Expected: add a peak_granted_at field, or label the column as the time of peak waiters.
W10. Rate columns show the latest minute under a "last 24 hours" title
On the Waits tab, the latch and spinlock panels are titled "last 24 hours". Their totals cover 24 hours, but their
rate columns cover only the latest minute:
- Latch
BUFFER: 31.5 minutes of wait, at 6.00 ms per second. - Spinlock
LOCK_RW_ABTX_HASH_SET: 1,378,602 collisions, at 0.00 per second.
The MCP tool description explains this. The web panel does not.
Expected: label the rate columns "latest minute", or compute them over the window.
W11. Job History and Collection Health rows are hundreds of pixels tall
- Job History: 58 of the 100 rows are taller than 120 px, and the page is 22,250 px long. The long MESSAGE
column is past the right edge and wraps in a narrow width. - Collection Health: 7 of the 41 collector rows are taller than 120 px, because of the NOTE and OUTPUT columns.
Expected: truncate long text cells to one line and show the rest on demand.
W12. A chart titled "last 24 hours" plots only a row-capped subset
On System Events, the "system_health CPU" chart is titled "last 24 hours" but plots only from about 6:31 AM. The
grid below it says "partial window: this grid shows only the newest rows, back to 10/9/2026, 6:31:32 AM, because it
stops at its row limit". The chart has no such notice.
The Active Queries grid is also titled "last 24 hours", but it shows only its 50-row limit, which covers about one
hour.
Expected: show the partial-window notice on every panel built from a capped read.
W13. Collection Health says an expected zero "needs a look"
For collectors whose zero result is the normal answer, the note says "a persistent zero here is not a resting
state: the source came back empty on every run, which on a target that has anything for it to report needs a
look". This shows for the AG states on a server with no availability groups, for trace flags with none enabled, and
for running jobs.
Expected: do not suggest action when the server has nothing to report for that source.
W14. On Alert History, the Triage and Mute actions start off screen (minor)
At 1600 px wide, the Alert History grid is 1,791 px wide inside a 1,298 px box. TRIAGE and MUTE, the two action
columns, are reachable only by horizontal scroll.
W15. Typo "latch classs"
The Waits tab says "a trend for the latch classs you check" and "Top latch classs". The noun is "latch class"
(wwwroot/js/pages/server-tabs.js:4077), and the picker adds a plain "s" to form the plural.
W16. The Blocking tab has two Deadlocks panels
The Blocking tab shows two panels titled "Deadlocks · last 24 hours". One says "No deadlock was recorded … 200
collector runs …". The other says "No deadlocks found in the specified time range".
W17. Restart banner lists an unchanged name as a change
The second server's Overview says: "instance identity changed — start time 2026-10-08 00:10:55 → 2026-10-09
00:10:55, server name → ", with the same name on both sides. The name did not change. Also, the start time moved by exactly
24 hours to the second. I did not confirm whether the server restarted at that time.
W18. FinOps verdicts disagree on the same server
At the same time, FinOps Utilization says "RIGHT SIZED" (P95 24%), and Recommendations says "CPU over-provisioned".
This extends M11.
W19. Labels that misdescribe their values (minor)
- "HT ratio 16" sits next to 1 socket with 8 cores. The value is logical CPUs per socket. The hyperthread ratio is 2.
- The Threads tile says "518 free", over "518 / 704 threads". That reads as 518 in use.
- FinOps Avg, P95 and Max CPU are SQL Server process CPU (
sqlserver_cpu_utilization). The header and Fleet CPU
are total CPU. Neither label says which. - The Memory tab shows "Plan cache 6.3 GB" (memory clerk) and "Cache size 3.0 GB" (plan cache detail), without
saying how they differ. - "% OF SWEEP" under Heaviest Collectors is the share of one run's budget (
pct_of_sweep_budget_per_run), not the
share of sweep time. - Significant Waits has the header "DURATION MS", but its values are shown in seconds, for example "32.9 s".
W20. Timestamps in several formats on one page (minor)
System Events and Triage show raw ISO strings in UTC, such as "2026-10-09T14:37:35.0158630" and
"2026-10-08T18:11:14Z", next to local-time grids such as "10/9/2026, 10:36:32 AM". The restart banner shows times
with no time zone.
W21. Repeated rows in two grids (minor)
- On Job History, LAST SUCCESS repeats the job's current last success on every row, so older rows show a time
later than their own run time. - On Queries, Plan Corrections lists the same 3 recommendations again every 5 minutes. All 50 rows are 3 query ids.
W22. A non-numeric hours value silently falls back to the default (minor, web API)
| Request | Result |
|---|---|
hours=-5 |
400: "Must be a positive integer (1-168)." |
hours=500 |
400: "exceeds maximum of 168 hours" |
hours=abc |
200, with hours_back 4, the default |
hours=1.5 on get_wait_stats |
200, with hours_back 24, the default |
QueryInt in DarlingWebEndpoints.cs (about line 6172) returns the default when int.TryParse fails.
Expected: refuse a non-integer value, as an out-of-range one is refused.
MCP defects that also show on the web
| MCP issue | Where it shows |
|---|---|
| M1 self-blocking counted | Alert History "Blocking chain (x58)", Blocking tab, Triage (blocker and blocked both session 59, lock mode "(none)", object "Unresolved: database: master"), Overview "Blocking 58" with a Warning band |
| M4 dropped databases | FinOps Index Analysis (four dropped scratch databases), Recommendations (a dropped database, dbo.r) |
| M7 monitor's own queries | Blocking grid, Queries tab (Query Store regressions, heatmap), System Events (Significant Waits) |
| M9 min server memory | Configuration tab: "min server memory (MB) 0 16 no" |
| M11 two CPU recommendations | Recommendations: about 5 cores at P95 24%, and 8 cores at P95 22% |
| M15 NULL-first sizes, collector temp tables | File I/O tab |
| M16 clustered "Unused" | Activity tab, FinOps Index Analysis |
| M18 waiting tasks | Waits tab: sessions blocked by themselves |
| M19 trace-flag reason | Config Changes tab |
| M20 $0 cost, two health scores | Admin Servers "$0" for all servers, Server Inventory; health 92 on Utilization and 91 on Inventory |
| M22 parallelism as CPU | FinOps Optimization: "CPU" category with CXCONSUMER as its top wait |
Checked, not a defect
- The web Fleet Overview and MCP
get_fleet_overviewagree. Both default to a 1-hour window. - Wide grids scroll horizontally, so the cut-off columns that the screenshot review flagged can be reached.
/api/server-databasesreturns 404 because it is newer than the installed build.
🤖 Generated with Claude Code
Steps to Reproduce
View web viewer
Expected Behavior
It should work, Erik
Actual Behavior
It doesn't work, Erik
Error Messages / Log Output
Screenshots
No response
Additional Context
No response
Activity
erikdarlingdata commentedon Oct 11, 2026
Claude posting for Erik Darling.
Triage notes for the agent who picks this up. Most of the 22 defects (W1 to W22) are confirmed or likely in code, and most are small. Nobody has changed any code yet.
Code read on
devat2d9709215. This was a code reading, with no runtime test.Three fixes landed after the audit build (10-06).
fd80b599e(#5554) addedJS/plain-text.js.f98ab412a(#5492) and25a9e0a03(#5586, chart width) also landed. Only #5554 touches a numbered defect, W8. Paths: SVC isDarling/PerformanceMonitor.Darling.Service, JS isSVC/wwwroot/js, COL isPerformanceMonitor.Collectors."Verified" means someone read the cited line. "Inferred" means it fits the code but is not proven.
PerformanceMonitor.Alerting/AlertEngine.cs:1047-1048counts one row per XE report. The fleet "Blocking N" tile is the same kind of count (SVC/Mcp/DarlingFleetReader.cs:1046and:1127)AlertEngine.cs:1048). The detail isGroupBlockingoverFilterBlockingoutput (AlertContextBuilders.cs:64-77and:502). The filter that drops the row was not foundCOL/CpuUtilizationCollector.cs:178-230computessample_timefromms_ticksandSYSDATETIME(). Dedup is client-side on that stamp (about lines 316-335). The query never reads the record idRecord/@idand the ring-buffer timestamp, and dedup on them. Or roundsample_timeto the ring-buffer tickDarlingFinOpsDatabaseResourcesReader.cs:366storesConvert.ToInt64(reader.GetValue(6))inCpuTimeMs. Line 380 ranks on it. 2.3, 1.95 and 1.56 all become 2avg_cpu_msas decimal. Rank on it. Round at displayCOL/PvsStatsCollector.cs:245-253divides bysys.master_filessize, which is the startup size for tempdbtempdb.sys.database_files. May touch a SQL-pinned testCOL/DefaultTraceEventsCollector.cs:228-285has no subclass column or filter.SVC/Mcp/DarlingDefaultTraceReader.cs:115-135has no DISTINCT. The begin and commit pair is from the audit and was not checkedJS/charts.jstreats null as a gap (about lines 285 and 508). Nothing inserts nulls where timestamps jumpplain-text.jsrewrites "see hints.captures" and "widen hours_back". Not covered: "pass bucket_minutes", "raise limit" (DarlingMcpObjectStatsTools.cs:302,305,519,DarlingMcpPgDatabaseTools.cs:330,341,DarlingMcpStoreMetricsTools.cs:679), "matched blocking_only" and "this is the healthy answer for this read" (DarlingMcpHealthParserTools.cs:762)plain-text.js.WebPlainTextTestspins them. Or reword at the sourceJS/pages/server-tabs.js:3901labelspeak_waiters_atas "Peak at". It sits besidepeak_granted_memory_mb(:3902)peak_granted_atto the payloadserver-tabs.jsaround lines 4170-4200. The rate comes from the readJS/pages/job-history.jsand the Collection Health table.util.js:90-103hastruncate()and a<details>helper that these cells seem not to useJS/util.js:283-289builds the partial-window notice for grids. The system_health CPU chart and the Active Queries panel do not carry itDarlingMcpDataTools.cs:1654-1664). Trace flags and running jobs still get the generic zero-output finding (FormatOutputFinding, near:2143)JS/pages/alerts.jscolumn order. The grid is wider than its boxserver-tabs.js:4178hasnoun: "latch class". Lines 4201 and 4216 build the plural asspec.noun + "s"pluralfield, for exampleplural: "latch classes"server-tabs.js:1325(trend chart) and:1345(table) both use "Deadlocks". The empty messages differCOL/BaselineDiscontinuity.cs:333-338always prints the start time and name pairFinOpsUtilizationFigures.cs:89says RIGHT_SIZED.FinOpsRecommendationFigures.cs:357says "CPU over-provisioned". The thresholds and the CPU basis differserver-tabs.js:2959showshyperthread_ratioas "HT ratio" but it holds logical CPUs per socket.:4926has "% of sweep" forpct_of_sweep_budget_per_run. Not read: threads "free", FinOps CPU basis, plan cache clerk, "DURATION MS"format: "time"job-history.js:55shows the job's currentlast_successon each row.server-tabs.js:1710runsget_plan_corrections limit 50, one row per observationhoursfalls back to the defaultSVC/DarlingWebEndpoints.cs:6390-6391(QueryInt) callsParseInt(:6475-6476), which returns the default whenint.TryParsefailsParseIntisinternaland used elsewhere, so add a strict variant. Run the web endpoint testsErik's ruling (W1 and W2)
Erik's ruling: count one blocking event per blocked session, blocker and wait start. Erik: "i agree with all recs".
Erik's call
These are wording or design choices, not bugs. Do not pick one without Erik.
peak_granted_atfield.Suggested lanes
plain-text.jsrules. One PR. Coordinate W13 with [BUG] MCP tool audit on 3.9.0-nightly.20261006.515: 23 defects (blocking, spills, FinOps, store, tool payloads) #5633, which edits the sameget_collection_healthtext.Re-check before coding
dev. Re-check on a current build.FinOps top-by-average ranks on the unrounded CPU average; Version Sto…
CPU collector no longer stores the newest ring-buffer record twice wh…
Web audit labels W9, W14, W16, W19: peak_granted_at, Triage/Mute firs…
Blocking events count waits, not blocked-process reports (#5644 W1, W2)
Pin the blocking wait count: one event per wait, SQL and C# agree (#5644
Web wording: caller-only clauses, latch/spinlock rate labels, plural,…
Default Trace: stop re-storing the newest event and keep the Commit r…
Web UI audit lane F: strict integer query params, changed-only restar…
Merge origin/fix/5644-web-ui-audit-lane-e into the web UI audit branch (
G1a: update pins the lane E merge made stale; test plainText and loca…
Pin the CPU ring-buffer dedup with its tolerance; run the DuckDB bloc…
G1a: relabel plan cache (all cache stores); name the server clock in …
17 remaining items
Default Trace readers keep two real events with identical columns; ev…
Count blocking waits, not report rows, on Lite's server tab badge and…
Starter dashboard blocking trend panel is titled for reports or snaps…
Overview tab Deadlocks chart uses the same empty sentence as the Bloc…
Tests follow the web UI audit: Alert History harness finds Metric by …
Update four test pins the #5644 web UI audit outdated: wait-counting …
CI fixes: four pins follow the #5644 wording and panel changes; darli…
Band blocking on report counts again; /api/fleet refuses out-of-range…
Band the daily summary and the fleet sweep on blocked-process report …
The fleet bounded-read reference query returns the band's report coun…
Pin that the daily summary SQL carries the skip collector's name (#5644)
Web UI audit fixes (#5644) (#5705)
erikdarlingdata commentedon Oct 13, 2026
Claude posting for Erik Darling.
Fixed by #5705, merged to dev as 3f3b9c1. W1-W22 are fixed except W18, which helper-6's #5704 carried (merged as 52ac1f5). The PR body's "Fixes by item" table maps every item to its change, and "Decisions and known limits" lists the calls made.
Follow-ups filed from this work:
default_trace_eventsmeasures count stored copies twice (PR Custom Views count each Default Trace event once (#5738) #5742).The 17 changelog entries are in the release buffer.