Skip to content

[BUG] Web UI audit on 3.9.0-nightly.20261006.515: 22 defects (blocking counts, CPU duplicates, FinOps, charts, UI wording) #5644

Closed

Description

@erikdarlingdata

Component

Darling

Performance Monitor Version

3.9.0

SQL Server Version

All of them

Windows Version

All of them

Describe the Bug

Web UI audit of a live Darling install. This issue lists 22 web defects (W1 to W22), and the MCP defects that also show on the web.

Build: 3.9.0-nightly.20261006.515, with the web listener on the LAN. Date: 2026-10-09. A separate MCP audit found 23
defects. They are referred to below as M1 to M23.

How the audit ran

  • Pages: a headless Chrome on Windows, with its own throwaway profile, driven over the DevTools protocol by
    a small script. It opened 42 hash routes:
    • the top-level pages
    • all 3 Admin tabs
    • all 12 server tabs for one test instance, plus one overview for a second server
    • all 12 FinOps tabs
    • one alert triage deep link
    • the three create forms, opened but never saved
  • Data saved for each page: a full-page screenshot, the visible text, every /api call, console errors,
    exceptions and failed requests. Nothing was clicked, so nothing was saved.
  • Positive control: a missing page, an unknown server and a missing view. The driver caught every 404 and 400
    with its console error, so an empty error list means no errors.
  • Visual review: four reviewers read 94 screenshot tiles. I checked every finding below against the page text,
    a DOM measurement, the store's API, the code or the SQL Server itself. Claims I could not confirm are left out.
  • Web API: the web-only routes were called with curl and a session cookie.
  • Not tested: the 21 routes that write. /api/server-databases returns 404 because it reached dev the same
    morning the nightly was built.

Result at a glance

  • All 42 pages loaded in 3.7 to 5.7 seconds, with no console errors, no exceptions and no failed requests.
  • No page showed NaN, undefined, null, Invalid Date or [object Object].
  • Every wide grid has a working horizontal scroller. Columns past the right edge can be reached, so they are not
    listed as defects, apart from W14.

New defects

W1. A blocking "event" is one blocked-process report, so one long wait counts many times

On the test day, one instance showed "Blocking 43", and alerts fired with values 37 and 43. Grouping the 43 reports by
blocked session, blocker and batch start gives only 12 distinct waits:

Blocked session Blocker Reports Wait time across the reports
102 100 11 5.9 s rising to 56.2 s
102 100 9 6.9 s rising to 52.0 s
102 100 9 7.1 s rising to 57.2 s
102 100 6 6.3 s rising to 31.4 s

The blocked process threshold is 5 seconds, so SQL Server reports a still-blocked session every 5 seconds. Every
report is counted as a new event, in the Blocking tile, the alert value and the analysis. A one-minute wait counts
as about 11 events. This is separate from M1, which is about a parallel query that blocks itself.

Expected: count one wait as one event (key on blocked session, blocker and wait start), and report its longest
wait time.

W2. The alert value and the alert detail differ by one

On Alert History, the value is one higher than the chain count in the detail for two rows:

Time Value Detail
10/9 9:19:15 AM 43.0 "Blocking chain (x42)"
10/9 9:13:56 AM 37.0 "Blocking chain (x36)"

The other blocking rows match, for example 58.0 with "(x58)".

W3. The CPU collector stores near-duplicate samples

sys.dm_os_ring_buffers on the test instance had exactly 60 scheduler-monitor records in the last hour, 60
seconds apart. The store has 82 to 87 samples in each full hour. In the last 61 minutes, 28 minutes have two
samples. The FinOps page shows this too: "1,593 CPU samples (24h)" on a server that was collected for less than
24 hours.

Cause: PerformanceMonitor.Collectors/CpuUtilizationCollector.cs (about line 179) re-reads the ring buffer
each poll. It computes sample_time from ms_ticks and SYSDATETIME(), then deduplicates on that computed value.
The comment there says that "a physical entry has no stable key". But each scheduler-monitor record carries a
Record id attribute, which is stable.

Expected: deduplicate on the record id. Duplicates inflate sample counts and percentiles, and they give extra
weight to the minutes that happen to be duplicated.

W4. FinOps "Top Databases by Avg CPU / Execution" leaves out the top database

On the FinOps page for one server, the busiest user database used 2,498,148 ms of CPU over 1,066,691 executions, which is
2.3 ms per execution. The "Avg CPU / Execution" table does not list it. It does list databases at 1.95 ms and
1.56 ms.

Cause: DarlingFinOpsDatabaseResourcesReader.GetTopResourceConsumersAsync (about line 366) converts
avg_cpu_ms with Convert.ToInt64 before ranking. 2.3, 1.95 and 1.56 all become 2. The tie then breaks on
database name, and the top 5 cut drops the busiest database.

Expected: rank on the unrounded average and round only for display.

W5. Version store panel shows tempdb's startup size, not its current size

The Version Store tab shows tempdb's "Database data size" as 32.0 GB. The Database Sizes tab shows 8 data files of
5.5 GB, which is 44 GB. On the server, sys.master_files gives 32,768 MB for tempdb, which is its startup size,
and tempdb.sys.database_files gives 45,056 MB, which is its current size.

Cause: PerformanceMonitor.Collectors/PvsStatsCollector.cs takes the size denominator from sys.master_files.

Expected: read tempdb's size from tempdb.sys.database_files.

W6. Default Trace shows every DDL event twice

On System Events, every DDL row appears twice with identical values. 58 of the 100 rows returned by MCP
get_default_trace_events are exact duplicates. On the server, the default trace has 3,280 Object:Created rows
with subclass 0 and 3,278 with subclass 1 in the last 24 hours. SQL Server writes one row when the operation
begins and one when it commits. The payload has no subclass column, so the two rows look the same. The duplicates
fill the row limit, and the grid covers only about 18 minutes.

Expected: keep only one subclass for object events, or return event_subclass.

W7. Charts draw straight lines across collection gaps

The service host is off every night. The CPU chart on the CPU tab draws a straight diagonal from about 11 PM to
6:30 AM, when no data was collected. The same happens in the Blocking charts, the tempdb write-latency chart, and
the second server's memory chart. On that memory chart, a restart shows as a smooth slide from about 98 GB to about
5 GB. A reader cannot tell the gap from real data.

Expected: break the line wherever no samples exist.

W8. Messages written for API callers appear in the UI

The pages show tool messages as written for an MCP caller. Examples:

  • "see hints.captures for which collectors ran and when" (Overview, Blocking, second server's Overview)
  • "pass bucket_minutes (1-1440) for another width, or narrow hours_back for finer points" (Waits, CPU, Memory,
    Activity)
  • "widen hours_back to find the most recent samples" (Queries)
  • "raise limit to see more" (FinOps Locking, which has no limit control)
  • "in the last 1 hour(s) matched blocking_only" (Triage)
  • "this is the healthy answer for this read" (System Events)

Expected: the web page should give its own wording, or name the page control that does the same thing.

W9. "PEAK AT" shows the wrong time for the peak memory grant

On the Memory tab, the peak granted memory of 86.5 GB has a "PEAK AT" time that is the newest snapshot. At that
snapshot, 0 MB was granted. The MCP payload has only peak_waiters_at, the time of the most waiters. With zero
waiters, that is the newest snapshot. The page shows it next to the peak grant.

Expected: add a peak_granted_at field, or label the column as the time of peak waiters.

W10. Rate columns show the latest minute under a "last 24 hours" title

On the Waits tab, the latch and spinlock panels are titled "last 24 hours". Their totals cover 24 hours, but their
rate columns cover only the latest minute:

  • Latch BUFFER: 31.5 minutes of wait, at 6.00 ms per second.
  • Spinlock LOCK_RW_ABTX_HASH_SET: 1,378,602 collisions, at 0.00 per second.

The MCP tool description explains this. The web panel does not.

Expected: label the rate columns "latest minute", or compute them over the window.

W11. Job History and Collection Health rows are hundreds of pixels tall

  • Job History: 58 of the 100 rows are taller than 120 px, and the page is 22,250 px long. The long MESSAGE
    column is past the right edge and wraps in a narrow width.
  • Collection Health: 7 of the 41 collector rows are taller than 120 px, because of the NOTE and OUTPUT columns.

Expected: truncate long text cells to one line and show the rest on demand.

W12. A chart titled "last 24 hours" plots only a row-capped subset

On System Events, the "system_health CPU" chart is titled "last 24 hours" but plots only from about 6:31 AM. The
grid below it says "partial window: this grid shows only the newest rows, back to 10/9/2026, 6:31:32 AM, because it
stops at its row limit". The chart has no such notice.

The Active Queries grid is also titled "last 24 hours", but it shows only its 50-row limit, which covers about one
hour.

Expected: show the partial-window notice on every panel built from a capped read.

W13. Collection Health says an expected zero "needs a look"

For collectors whose zero result is the normal answer, the note says "a persistent zero here is not a resting
state: the source came back empty on every run, which on a target that has anything for it to report needs a
look". This shows for the AG states on a server with no availability groups, for trace flags with none enabled, and
for running jobs.

Expected: do not suggest action when the server has nothing to report for that source.

W14. On Alert History, the Triage and Mute actions start off screen (minor)

At 1600 px wide, the Alert History grid is 1,791 px wide inside a 1,298 px box. TRIAGE and MUTE, the two action
columns, are reachable only by horizontal scroll.

W15. Typo "latch classs"

The Waits tab says "a trend for the latch classs you check" and "Top latch classs". The noun is "latch class"
(wwwroot/js/pages/server-tabs.js:4077), and the picker adds a plain "s" to form the plural.

W16. The Blocking tab has two Deadlocks panels

The Blocking tab shows two panels titled "Deadlocks · last 24 hours". One says "No deadlock was recorded … 200
collector runs …". The other says "No deadlocks found in the specified time range".

W17. Restart banner lists an unchanged name as a change

The second server's Overview says: "instance identity changed — start time 2026-10-08 00:10:55 → 2026-10-09
00:10:55, server name → ", with the same name on both sides. The name did not change. Also, the start time moved by exactly
24 hours to the second. I did not confirm whether the server restarted at that time.

W18. FinOps verdicts disagree on the same server

At the same time, FinOps Utilization says "RIGHT SIZED" (P95 24%), and Recommendations says "CPU over-provisioned".
This extends M11.

W19. Labels that misdescribe their values (minor)

  • "HT ratio 16" sits next to 1 socket with 8 cores. The value is logical CPUs per socket. The hyperthread ratio is 2.
  • The Threads tile says "518 free", over "518 / 704 threads". That reads as 518 in use.
  • FinOps Avg, P95 and Max CPU are SQL Server process CPU (sqlserver_cpu_utilization). The header and Fleet CPU
    are total CPU. Neither label says which.
  • The Memory tab shows "Plan cache 6.3 GB" (memory clerk) and "Cache size 3.0 GB" (plan cache detail), without
    saying how they differ.
  • "% OF SWEEP" under Heaviest Collectors is the share of one run's budget (pct_of_sweep_budget_per_run), not the
    share of sweep time.
  • Significant Waits has the header "DURATION MS", but its values are shown in seconds, for example "32.9 s".

W20. Timestamps in several formats on one page (minor)

System Events and Triage show raw ISO strings in UTC, such as "2026-10-09T14:37:35.0158630" and
"2026-10-08T18:11:14Z", next to local-time grids such as "10/9/2026, 10:36:32 AM". The restart banner shows times
with no time zone.

W21. Repeated rows in two grids (minor)

  • On Job History, LAST SUCCESS repeats the job's current last success on every row, so older rows show a time
    later than their own run time.
  • On Queries, Plan Corrections lists the same 3 recommendations again every 5 minutes. All 50 rows are 3 query ids.

W22. A non-numeric hours value silently falls back to the default (minor, web API)

Request Result
hours=-5 400: "Must be a positive integer (1-168)."
hours=500 400: "exceeds maximum of 168 hours"
hours=abc 200, with hours_back 4, the default
hours=1.5 on get_wait_stats 200, with hours_back 24, the default

QueryInt in DarlingWebEndpoints.cs (about line 6172) returns the default when int.TryParse fails.

Expected: refuse a non-integer value, as an out-of-range one is refused.

MCP defects that also show on the web

MCP issue Where it shows
M1 self-blocking counted Alert History "Blocking chain (x58)", Blocking tab, Triage (blocker and blocked both session 59, lock mode "(none)", object "Unresolved: database: master"), Overview "Blocking 58" with a Warning band
M4 dropped databases FinOps Index Analysis (four dropped scratch databases), Recommendations (a dropped database, dbo.r)
M7 monitor's own queries Blocking grid, Queries tab (Query Store regressions, heatmap), System Events (Significant Waits)
M9 min server memory Configuration tab: "min server memory (MB) 0 16 no"
M11 two CPU recommendations Recommendations: about 5 cores at P95 24%, and 8 cores at P95 22%
M15 NULL-first sizes, collector temp tables File I/O tab
M16 clustered "Unused" Activity tab, FinOps Index Analysis
M18 waiting tasks Waits tab: sessions blocked by themselves
M19 trace-flag reason Config Changes tab
M20 $0 cost, two health scores Admin Servers "$0" for all servers, Server Inventory; health 92 on Utilization and 91 on Inventory
M22 parallelism as CPU FinOps Optimization: "CPU" category with CXCONSUMER as its top wait

Checked, not a defect

  • The web Fleet Overview and MCP get_fleet_overview agree. Both default to a 1-hour window.
  • Wide grids scroll horizontally, so the cut-off columns that the screenshot review flagged can be reached.
  • /api/server-databases returns 404 because it is newer than the installed build.

🤖 Generated with Claude Code

Steps to Reproduce

View web viewer

Expected Behavior

It should work, Erik

Actual Behavior

It doesn't work, Erik

Error Messages / Log Output


Screenshots

No response

Additional Context

No response

Activity

  1. erikdarlingdata commented on Oct 11, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Triage notes for the agent who picks this up. Most of the 22 defects (W1 to W22) are confirmed or likely in code, and most are small. Nobody has changed any code yet.

    Code read on dev at 2d9709215. This was a code reading, with no runtime test.

    Three fixes landed after the audit build (10-06). fd80b599e (#5554) added JS/plain-text.js. f98ab412a (#5492) and 25a9e0a03 (#5586, chart width) also landed. Only #5554 touches a numbered defect, W8. Paths: SVC is Darling/PerformanceMonitor.Darling.Service, JS is SVC/wwwroot/js, COL is PerformanceMonitor.Collectors.

    "Verified" means someone read the cited line. "Inferred" means it fits the code but is not proven.

    # Defect Verdict Cause Likely fix Size
    W1 Blocking counted per blocked-process report CONFIRMED PerformanceMonitor.Alerting/AlertEngine.cs:1047-1048 counts one row per XE report. The fleet "Blocking N" tile is the same kind of count (SVC/Mcp/DarlingFleetReader.cs:1046 and :1127) Count one event per blocked session, blocker and wait start. Report the longest wait. Use it in the engine, the fleet tile and the analysis M to L
    W2 Alert value is detail count plus 1 LIKELY (inferred) The value is the raw row count (AlertEngine.cs:1048). The detail is GroupBlocking over FilterBlocking output (AlertContextBuilders.cs:64-77 and :502). The filter that drops the row was not found Take the alert value from the same filtered and grouped set as the detail S
    W3 CPU collector near-duplicate samples CONFIRMED (design). The duplicates were not reproduced COL/CpuUtilizationCollector.cs:178-230 computes sample_time from ms_ticks and SYSDATETIME(). Dedup is client-side on that stamp (about lines 316-335). The query never reads the record id Select Record/@id and the ring-buffer timestamp, and dedup on them. Or round sample_time to the ring-buffer tick M
    W4 FinOps Avg CPU per Execution drops the top database CONFIRMED DarlingFinOpsDatabaseResourcesReader.cs:366 stores Convert.ToInt64(reader.GetValue(6)) in CpuTimeMs. Line 380 ranks on it. 2.3, 1.95 and 1.56 all become 2 Carry avg_cpu_ms as decimal. Rank on it. Round at display S
    W5 Version store panel uses tempdb startup size CONFIRMED COL/PvsStatsCollector.cs:245-253 divides by sys.master_files size, which is the startup size for tempdb For database id 2, read tempdb.sys.database_files. May touch a SQL-pinned test S
    W6 Default trace DDL rows twice LIKELY (inferred) COL/DefaultTraceEventsCollector.cs:228-285 has no subclass column or filter. SVC/Mcp/DarlingDefaultTraceReader.cs:115-135 has no DISTINCT. The begin and commit pair is from the audit and was not checked Keep one subclass in the collector, or dedup in the reader. A new column needs a migration rung M
    W7 Charts draw lines across collection gaps LIKELY (inferred from absence of code) JS/charts.js treats null as a gap (about lines 285 and 508). Nothing inserts nulls where timestamps jump Insert a null point when the gap exceeds a multiple of the median interval. One central fix covers all charts M
    W8 MCP-caller wording in the UI PARTLY FIXED by #5554. The rest is CONFIRMED plain-text.js rewrites "see hints.captures" and "widen hours_back". Not covered: "pass bucket_minutes", "raise limit" (DarlingMcpObjectStatsTools.cs:302,305,519, DarlingMcpPgDatabaseTools.cs:330,341, DarlingMcpStoreMetricsTools.cs:679), "matched blocking_only" and "this is the healthy answer for this read" (DarlingMcpHealthParserTools.cs:762) Add rules to plain-text.js. WebPlainTextTests pins them. Or reword at the source S
    W9 PEAK AT is the peak-waiters time CONFIRMED JS/pages/server-tabs.js:3901 labels peak_waiters_at as "Peak at". It sits beside peak_granted_memory_mb (:3902) Relabel "Peak waiters at", or add peak_granted_at to the payload S or M
    W10 Rate columns show the latest minute under "last 24 hours" NEEDS RUNTIME. The code was not read The latch and spinlock panels in server-tabs.js around lines 4170-4200. The rate comes from the read Relabel "latest minute", or compute over the window S to M
    W11 Job History and Collection Health rows too tall LIKELY (inferred) JS/pages/job-history.js and the Collection Health table. util.js:90-103 has truncate() and a <details> helper that these cells seem not to use Use the helper for message, note and output columns S
    W12 "last 24 hours" chart or grid is a row-capped subset LIKELY (inferred) JS/util.js:283-289 builds the partial-window notice for grids. The system_health CPU chart and the Active Queries panel do not carry it Pass the notice to chart panels and capped grids S to M
    W13 "needs a look" on an expected zero PARTLY FIXED. The rest is CONFIRMED The availability group collectors are fixed (DarlingMcpDataTools.cs:1654-1664). Trace flags and running jobs still get the generic zero-output finding (FormatOutputFinding, near :2143) Add trace flags and running jobs to the known-zero list S
    W14 Alert History actions off screen LIKELY JS/pages/alerts.js column order. The grid is wider than its box See "Erik's call" S
    W15 "latch classs" CONFIRMED server-tabs.js:4178 has noun: "latch class". Lines 4201 and 4216 build the plural as spec.noun + "s" Add a plural field, for example plural: "latch classes" S
    W16 Two "Deadlocks" panels on the Blocking tab CONFIRMED (duplicate title) server-tabs.js:1325 (trend chart) and :1345 (table) both use "Deadlocks". The empty messages differ See "Erik's call" S
    W17 Restart banner lists an unchanged name as a change CONFIRMED COL/BaselineDiscontinuity.cs:333-338 always prints the start time and name pair Print only the sides that differ. Check separately why the start time moves by exactly 24 hours (NEEDS RUNTIME) S
    W18 FinOps verdicts disagree LIKELY (inferred) FinOpsUtilizationFigures.cs:89 says RIGHT_SIZED. FinOpsRecommendationFigures.cs:357 says "CPU over-provisioned". The thresholds and the CPU basis differ One shared rule for "over-provisioned". Same root as #5633-11 M
    W19 Misleading labels CONFIRMED (HT ratio, "% of sweep"). The others are LIKELY server-tabs.js:2959 shows hyperthread_ratio as "HT ratio" but it holds logical CPUs per socket. :4926 has "% of sweep" for pct_of_sweep_budget_per_run. Not read: threads "free", FinOps CPU basis, plan cache clerk, "DURATION MS" See "Erik's call" S each
    W20 Mixed timestamp formats LIKELY Raw ISO strings pass through where a column has no format: "time" See "Erik's call" S to M
    W21 Repeated rows (job LAST SUCCESS, plan corrections) LIKELY job-history.js:55 shows the job's current last_success on each row. server-tabs.js:1710 runs get_plan_corrections limit 50, one row per observation Drop or relabel LAST SUCCESS. Collapse plan corrections by query id S to M
    W22 Non-numeric hours falls back to the default CONFIRMED SVC/DarlingWebEndpoints.cs:6390-6391 (QueryInt) calls ParseInt (:6475-6476), which returns the default when int.TryParse fails Return 400 for a present non-integer. ParseInt is internal and used elsewhere, so add a strict variant. Run the web endpoint tests S

    Erik's ruling (W1 and W2)

    Erik's ruling: count one blocking event per blocked session, blocker and wait start. Erik: "i agree with all recs".

    Erik's call

    These are wording or design choices, not bugs. Do not pick one without Erik.

    • W9: relabel the column "Peak waiters at", or add a new peak_granted_at field.
    • W14: move Triage and Mute to the first column, or pin them.
    • W16: retitle the chart "Deadlocks trend", or merge the two empty messages.
    • W19: the wording for each misleading label.
    • W20: which time zone convention to use, and which columns to format.

    Suggested lanes

    1. Blocking count (W1, W2): the engine row count, the fleet tile and the analysis. Define "one wait" once.
    2. Small SQL and rank bugs (W4, W5, W3): one PR each. W3 touches the shared collector and Lite. W5 touches a SQL pin test.
    3. Web wording and labels (W8, W10, W13, W15, W9, W16, W19 after Erik's call): all JS plus plain-text.js rules. One PR. Coordinate W13 with [BUG] MCP tool audit on 3.9.0-nightly.20261006.515: 23 defects (blocking, spills, FinOps, store, tool payloads) #5633, which edits the same get_collection_health text.
    4. Web layout and charts (W7, W11, W12, W14, W20, W21): W7 is the only M item.
    5. Single fixes: W17, W22, and W6 (runtime check first).
    6. W18 with [BUG] MCP tool audit on 3.9.0-nightly.20261006.515: 23 defects (blocking, spills, FinOps, store, tool payloads) #5633-11.

    Re-check before coding

    • W8 and W13: partly fixed on dev. Re-check on a current build.
    • W2: the extra row may be an intended filter difference. The dropped row was not found.
    • W3: the audit's claim that the ring buffer record id is stable across polls was not checked.
    • W6: the begin and commit pair is the audit's measurement. Not checked here.
    • W10, W17 (24 hour shift) and W18 thresholds: need runtime or a longer read.
    • W10, W11, W12, W14, W20 and W21: located by name only. The panel specs were not read.
  2. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 12, 2026
  3. 17 remaining items

  4. erikdarlingdata commented on Oct 13, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Fixed by #5705, merged to dev as 3f3b9c1. W1-W22 are fixed except W18, which helper-6's #5704 carried (merged as 52ac1f5). The PR body's "Fixes by item" table maps every item to its change, and "Decisions and known limits" lists the calls made.

    Follow-ups filed from this work:

    The 17 changelog entries are in the release buffer.

  5. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions